[JBoss JIRA] (AS7-4932) CLONE - Using session passivation results in WeldListener: java.lang.NullPointerException on normal operation
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/AS7-4932?page=com.atlassian.jira.plugin.s... ]
RH Bugzilla Integration commented on AS7-4932:
----------------------------------------------
Paul Ferraro <paul.ferraro(a)redhat.com> changed the Status of [bug 900549|https://bugzilla.redhat.com/show_bug.cgi?id=900549] from ASSIGNED to MODIFIED
> CLONE - Using session passivation results in WeldListener: java.lang.NullPointerException on normal operation
> -------------------------------------------------------------------------------------------------------------
>
> Key: AS7-4932
> URL: https://issues.jboss.org/browse/AS7-4932
> Project: Application Server 7
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 7.1.2.Final (EAP)
> Reporter: Radoslav Husar
> Assignee: Paul Ferraro
> Priority: Blocker
> Labels: as713tracking
> Fix For: 7.1.3.Final (EAP)
>
>
> NPE is thrown in roughly 0.36% of HTTP session request processing in the test.
> This results in response code 503 returned to the client with the exception.
> The test is using passivation-enabled WAR of clusterbench
> https://github.com/rhusar/clusterbench
> Here is a shorter soak test run that uncovered the issue
> https://hudson.qa.jboss.com/hudson/view/EAP6/view/EAP6-Clustering-Soak/jo...
> {noformat}
> [JBossINF] 19:52:10,113 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host]] (ajp-perf20/10.16.90.58:8009-2570) Exception sending request initialized lifecycle event to listener instance of class org.jboss.weld.servlet.WeldListener: java.lang.NullPointerException
> [JBossINF] at org.jboss.as.web.session.ClusteredSession.update(ClusteredSession.java:972) [jboss-as-web-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
> [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.loadSession(DistributableSessionManager.java:1377) [jboss-as-web-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
> [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:673) [jboss-as-web-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
> [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:84) [jboss-as-web-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
> [JBossINF] at org.apache.catalina.connector.Request.doGetSession(Request.java:2618) [jbossweb-7.0.16.Final-redhat-1.jar:]
> [JBossINF] at org.apache.catalina.connector.Request.getSession(Request.java:2375) [jbossweb-7.0.16.Final-redhat-1.jar:]
> [JBossINF] at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:841) [jbossweb-7.0.16.Final-redhat-1.jar:]
> [JBossINF] at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.getSession(LazySessionBeanStore.java:72) [weld-core-1.1.8.Final-redhat-1.jar:1.1.8.Final-redhat-1]
> [JBossINF] at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.<init>(LazySessionBeanStore.java:58) [weld-core-1.1.8.Final-redhat-1.jar:1.1.8.Final-redhat-1]
> [JBossINF] at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:31) [weld-core-1.1.8.Final-redhat-1.jar:1.1.8.Final-redhat-1]
> [JBossINF] at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:16) [weld-core-1.1.8.Final-redhat-1.jar:1.1.8.Final-redhat-1]
> [JBossINF] at org.jboss.weld.servlet.WeldListener.requestInitialized(WeldListener.java:134) [weld-core-1.1.8.Final-redhat-1.jar:1.1.8.Final-redhat-1]
> [JBossINF] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [jbossweb-7.0.16.Final-redhat-1.jar:]
> [JBossINF] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.16.Final-redhat-1.jar:]
> [JBossINF] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.16.Final-redhat-1.jar:]
> [JBossINF] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.16.Final-redhat-1.jar:]
> [JBossINF] at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.16.Final-redhat-1.jar:]
> [JBossINF] at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:452) [jbossweb-7.0.16.Final-redhat-1.jar:]
> [JBossINF] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931) [jbossweb-7.0.16.Final-redhat-1.jar:]
> [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
[JBoss JIRA] (WFLY-1271) Infinispan remote-store requires remote-servers but not marked as required in mgmt model
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/WFLY-1271?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated WFLY-1271:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=956904
> Infinispan remote-store requires remote-servers but not marked as required in mgmt model
> ----------------------------------------------------------------------------------------
>
> Key: WFLY-1271
> URL: https://issues.jboss.org/browse/WFLY-1271
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Takayoshi Kimura
> Assignee: Paul Ferraro
>
> Affect version is actually 8.0.0.Alpha1-SNAPSHOT (before 8.0.0.Alpha1, we don't have such selection).
> Command remote-store=REMOTE_STORE:add fails with this exception:
> {quote}
> 15:58:05,112 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 3) JBAS014607: Failed to persist configuration change: org.jboss.as.controller.persistence.ConfigurationPersistenceException: JBAS014675: Failed to marshal configuration
> at org.jboss.as.controller.persistence.AbstractFilePersistenceResource.<init>(AbstractFilePersistenceResource.java:50) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.persistence.ConfigurationFilePersistenceResource.<init>(ConfigurationFilePersistenceResource.java:45) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.persistence.BackupXmlConfigurationPersister.store(BackupXmlConfigurationPersister.java:80) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.ModelControllerImpl.writeModel(ModelControllerImpl.java:522) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.OperationContextImpl.createPersistenceResource(OperationContextImpl.java:178) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:359) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:228) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.AbstractOperationContext.finishStep(AbstractOperationContext.java:513) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:499) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:321) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:228) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:223) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:235) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:124) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:148) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:97) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:114) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:296) [jboss-as-protocol-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:518) [jboss-as-protocol-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_21]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_21]
> at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_21]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final.jar:2.1.0.Final]
> Caused by: org.jboss.as.controller.persistence.ConfigurationPersistenceException: JBAS014680: Failed to write configuration
> at org.jboss.as.controller.persistence.AbstractConfigurationPersister.marshallAsXml(AbstractConfigurationPersister.java:123) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.controller.persistence.AbstractFilePersistenceResource.<init>(AbstractFilePersistenceResource.java:43) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> ... 22 more
> Caused by: java.lang.IllegalArgumentException
> at org.jboss.dmr.ModelValue.asList(ModelValue.java:128) [jboss-dmr-1.1.6.Final.jar:1.1.6.Final]
> at org.jboss.dmr.ModelNode.asList(ModelNode.java:1205) [jboss-dmr-1.1.6.Final.jar:1.1.6.Final]
> at org.jboss.as.clustering.infinispan.subsystem.InfinispanSubsystemXMLWriter.processCommonCacheAttributesElements(InfinispanSubsystemXMLWriter.java:282)
> at org.jboss.as.clustering.infinispan.subsystem.InfinispanSubsystemXMLWriter.writeContent(InfinispanSubsystemXMLWriter.java:124)
> at org.jboss.as.clustering.infinispan.subsystem.InfinispanSubsystemXMLWriter.writeContent(InfinispanSubsystemXMLWriter.java:42)
> at org.jboss.as.server.parsing.StandaloneXml.writeServerProfile(StandaloneXml.java:1191) [jboss-as-server-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.server.parsing.StandaloneXml.writeContent(StandaloneXml.java:1108) [jboss-as-server-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.server.parsing.StandaloneXml.writeContent(StandaloneXml.java:103) [jboss-as-server-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> at org.jboss.staxmapper.XMLMapperImpl.doDeparse(XMLMapperImpl.java:88) [staxmapper-1.1.0.Final.jar:1.1.0.Final]
> at org.jboss.staxmapper.XMLMapperImpl.deparseDocument(XMLMapperImpl.java:83) [staxmapper-1.1.0.Final.jar:1.1.0.Final]
> at org.jboss.as.controller.persistence.AbstractConfigurationPersister.marshallAsXml(AbstractConfigurationPersister.java:117) [jboss-as-controller-8.0.0.Alpha1-SNAPSHOT.jar:8.0.0.Alpha1-SNAPSHOT]
> ... 23 more
> {quote}
> Reproduce steps:
> {quote}
> [standalone@localhost:9999 /] /subsystem=infinispan/cache-container=web/replicated-cache=repl/file-store=FILE_STORE:remove
> {
> "outcome" => "success",
> "response-headers" => {
> "operation-requires-reload" => true,
> "process-state" => "reload-required"
> }
> }
> [standalone@localhost:9999 /] /subsystem=infinispan/cache-container=web/replicated-cache=repl/remote-store=REMOTE_STORE:add
> {
> "outcome" => "failed",
> "failure-description" => "JBAS014677: Failed to persist configuration change: JBAS014675: Failed to marshal configuration",
> "rolled-back" => true,
> "response-headers" => {"process-state" => "reload-required"}
> }
> [standalone@localhost:9999 /] /subsystem=infinispan/cache-container=web/replicated-cache=repl/remote-store=REMOTE_STORE:add(remote-servers=[])
> {
> "outcome" => "success",
> "response-headers" => {"process-state" => "reload-required"}
> }
> {quote}
> In $JBOSS_HOME/docs/schema/jboss-as-infinispan_1_3.xsd, it looks like the remote-server is required as it's defined as minOccurs="1":
> {quote}
> <xs:element name="remote-server" type="tns:remote-server" minOccurs="1" maxOccurs="unbounded"/>
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
[JBoss JIRA] (JGRP-1618) missingMessageReceived() never called in NAKACK resulting in memory leak
by Harry Mark (JIRA)
[ https://issues.jboss.org/browse/JGRP-1618?page=com.atlassian.jira.plugin.... ]
Harry Mark edited comment on JGRP-1618 at 4/25/13 6:16 PM:
-----------------------------------------------------------
Bela, thanks for looking into this.
To track it down, I had to put in a lot of print statements into the code, run load tests across multiple servers, and analyze the logs. That's when I was able to identify the problem, which only shows up when there's enough load to cause UDP messages to arrive out of order.
To deploy the fix, I created a jar with the patched NakReceiverWindow, which is loaded before the jgroups jar. I have attached it. This patch is only for JGroups 2.8.1.
was (Author: hmark):
Bela, thanks for looking into this.
To track it down, I had to put in a lot of print statements into the code, run load tests across multiple servers, and analyze the logs. That's when I was able to identify the problem, which only shows up when there's enough load to cause UDP messages to arrive out of order.
To deploy the fix, I created a jar with the patched NakReceiverWindow, which is loaded before the jgroups 2.8.1 jar. I have attached it.
> missingMessageReceived() never called in NAKACK resulting in memory leak
> ------------------------------------------------------------------------
>
> Key: JGRP-1618
> URL: https://issues.jboss.org/browse/JGRP-1618
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.8.1, 2.12.2
> Environment: Java 6
> Reporter: Harry Mark
> Assignee: Bela Ban
> Fix For: 3.2.9, 3.3
>
> Attachments: 1618-jgrp-patch.jar, NakReceiverWindow.java
>
>
> We are using JGroups 2.8.1 and encountered a memory leak where it eventually ran out of CMS Old Gen memory. The heap dump revealed that the problem was in the xmit_stats ConcurrentHashMap of org.jgroups.protocols.pbcast.NAKACK.
> After much analysis here's what we found: when the system is under load, messages can start arriving out of order. When the receiver receives a higher sequence number than expected, it requests the sender retransmit the missing messages with the lower sequence numbers. The sender sends the missing message, however the bug in the NakReceiverWindow meant that the missing message was never purged from the Map that tracks missing messages (xmit_stats) because missingMessageReceived() was never invoked. Over time this Map grows and starts using up CMS Old Gen; the only way it would get reduced was when a server left the cluster and the missing messages were purged for that server.
> In JMX, the MissingMsgsReceived attribute of jgroups:cluster=*,protocol=NAKACK,type=protocol was always zero, confirming that it never purged any received "missing messages".
> When I looked at the most recent GA version 3.2.8 of NakReceiverWindow.java , it has the corrected logic that ensures that missingMessageReceived() is called. I checked the the most recent 2.x, which is 2.12.2, and found it also has the same bug in the logic as 2.8.1. This bug may apply to other 2.x but I did not check.
> Attached is the fixed NakReceiverWindow.java for 2.8.1.
> After applying the patch, the memory leak went away.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
[JBoss JIRA] (JGRP-1618) missingMessageReceived() never called in NAKACK resulting in memory leak
by Harry Mark (JIRA)
[ https://issues.jboss.org/browse/JGRP-1618?page=com.atlassian.jira.plugin.... ]
Harry Mark edited comment on JGRP-1618 at 4/25/13 6:13 PM:
-----------------------------------------------------------
Bela, thanks for looking into this.
To track it down, I had to put in a lot of print statements into the code, run load tests across multiple servers, and analyze the logs. That's when I was able to identify the problem, which only shows up when there's enough load to cause UDP messages to arrive out of order.
To deploy the fix, I created a jar with the patched NakReceiverWindow, which is loaded before the jgroups 2.8.1 jar. I have attached it.
was (Author: hmark):
Bela, thanks for looking into this.
To track it down, I had to put in a lot of print statements into the code, run load tests across multiple servers, and analyze the logs. That's when I was able to identify the problem, which only shows up when there's enough load to cause UDP messages to arrive out of order.
To deploy the fix, I created a jar with the patched NakReceiverWindow, which is loaded before the jgroups jar. I have attached it.
> missingMessageReceived() never called in NAKACK resulting in memory leak
> ------------------------------------------------------------------------
>
> Key: JGRP-1618
> URL: https://issues.jboss.org/browse/JGRP-1618
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.8.1, 2.12.2
> Environment: Java 6
> Reporter: Harry Mark
> Assignee: Bela Ban
> Fix For: 3.2.9, 3.3
>
> Attachments: 1618-jgrp-patch.jar, NakReceiverWindow.java
>
>
> We are using JGroups 2.8.1 and encountered a memory leak where it eventually ran out of CMS Old Gen memory. The heap dump revealed that the problem was in the xmit_stats ConcurrentHashMap of org.jgroups.protocols.pbcast.NAKACK.
> After much analysis here's what we found: when the system is under load, messages can start arriving out of order. When the receiver receives a higher sequence number than expected, it requests the sender retransmit the missing messages with the lower sequence numbers. The sender sends the missing message, however the bug in the NakReceiverWindow meant that the missing message was never purged from the Map that tracks missing messages (xmit_stats) because missingMessageReceived() was never invoked. Over time this Map grows and starts using up CMS Old Gen; the only way it would get reduced was when a server left the cluster and the missing messages were purged for that server.
> In JMX, the MissingMsgsReceived attribute of jgroups:cluster=*,protocol=NAKACK,type=protocol was always zero, confirming that it never purged any received "missing messages".
> When I looked at the most recent GA version 3.2.8 of NakReceiverWindow.java , it has the corrected logic that ensures that missingMessageReceived() is called. I checked the the most recent 2.x, which is 2.12.2, and found it also has the same bug in the logic as 2.8.1. This bug may apply to other 2.x but I did not check.
> Attached is the fixed NakReceiverWindow.java for 2.8.1.
> After applying the patch, the memory leak went away.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
[JBoss JIRA] (JGRP-1618) missingMessageReceived() never called in NAKACK resulting in memory leak
by Harry Mark (JIRA)
[ https://issues.jboss.org/browse/JGRP-1618?page=com.atlassian.jira.plugin.... ]
Harry Mark commented on JGRP-1618:
----------------------------------
Bela, thanks for looking into this.
To track it down, I had to put in a lot of print statements into the code, run load tests across multiple servers, and analyze the logs. That's when I was able to identify the problem, which only shows up when there's enough load to cause UDP messages to arrive out of order.
To deploy the fix, I created a jar with the patched NakReceiverWindow, which is loaded before the jgroups jar. I have attached it.
> missingMessageReceived() never called in NAKACK resulting in memory leak
> ------------------------------------------------------------------------
>
> Key: JGRP-1618
> URL: https://issues.jboss.org/browse/JGRP-1618
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.8.1, 2.12.2
> Environment: Java 6
> Reporter: Harry Mark
> Assignee: Bela Ban
> Fix For: 3.2.9, 3.3
>
> Attachments: 1618-jgrp-patch.jar, NakReceiverWindow.java
>
>
> We are using JGroups 2.8.1 and encountered a memory leak where it eventually ran out of CMS Old Gen memory. The heap dump revealed that the problem was in the xmit_stats ConcurrentHashMap of org.jgroups.protocols.pbcast.NAKACK.
> After much analysis here's what we found: when the system is under load, messages can start arriving out of order. When the receiver receives a higher sequence number than expected, it requests the sender retransmit the missing messages with the lower sequence numbers. The sender sends the missing message, however the bug in the NakReceiverWindow meant that the missing message was never purged from the Map that tracks missing messages (xmit_stats) because missingMessageReceived() was never invoked. Over time this Map grows and starts using up CMS Old Gen; the only way it would get reduced was when a server left the cluster and the missing messages were purged for that server.
> In JMX, the MissingMsgsReceived attribute of jgroups:cluster=*,protocol=NAKACK,type=protocol was always zero, confirming that it never purged any received "missing messages".
> When I looked at the most recent GA version 3.2.8 of NakReceiverWindow.java , it has the corrected logic that ensures that missingMessageReceived() is called. I checked the the most recent 2.x, which is 2.12.2, and found it also has the same bug in the logic as 2.8.1. This bug may apply to other 2.x but I did not check.
> Attached is the fixed NakReceiverWindow.java for 2.8.1.
> After applying the patch, the memory leak went away.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
[JBoss JIRA] (JGRP-1618) missingMessageReceived() never called in NAKACK resulting in memory leak
by Harry Mark (JIRA)
[ https://issues.jboss.org/browse/JGRP-1618?page=com.atlassian.jira.plugin.... ]
Harry Mark updated JGRP-1618:
-----------------------------
Attachment: 1618-jgrp-patch.jar
> missingMessageReceived() never called in NAKACK resulting in memory leak
> ------------------------------------------------------------------------
>
> Key: JGRP-1618
> URL: https://issues.jboss.org/browse/JGRP-1618
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.8.1, 2.12.2
> Environment: Java 6
> Reporter: Harry Mark
> Assignee: Bela Ban
> Fix For: 3.2.9, 3.3
>
> Attachments: 1618-jgrp-patch.jar, NakReceiverWindow.java
>
>
> We are using JGroups 2.8.1 and encountered a memory leak where it eventually ran out of CMS Old Gen memory. The heap dump revealed that the problem was in the xmit_stats ConcurrentHashMap of org.jgroups.protocols.pbcast.NAKACK.
> After much analysis here's what we found: when the system is under load, messages can start arriving out of order. When the receiver receives a higher sequence number than expected, it requests the sender retransmit the missing messages with the lower sequence numbers. The sender sends the missing message, however the bug in the NakReceiverWindow meant that the missing message was never purged from the Map that tracks missing messages (xmit_stats) because missingMessageReceived() was never invoked. Over time this Map grows and starts using up CMS Old Gen; the only way it would get reduced was when a server left the cluster and the missing messages were purged for that server.
> In JMX, the MissingMsgsReceived attribute of jgroups:cluster=*,protocol=NAKACK,type=protocol was always zero, confirming that it never purged any received "missing messages".
> When I looked at the most recent GA version 3.2.8 of NakReceiverWindow.java , it has the corrected logic that ensures that missingMessageReceived() is called. I checked the the most recent 2.x, which is 2.12.2, and found it also has the same bug in the logic as 2.8.1. This bug may apply to other 2.x but I did not check.
> Attached is the fixed NakReceiverWindow.java for 2.8.1.
> After applying the patch, the memory leak went away.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
[JBoss JIRA] (WFLY-82) Wrong injection made by container
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/WFLY-82?page=com.atlassian.jira.plugin.sy... ]
RH Bugzilla Integration commented on WFLY-82:
---------------------------------------------
Paul Gier <pgier(a)redhat.com> changed the Status of [bug 909914|https://bugzilla.redhat.com/show_bug.cgi?id=909914] from POST to MODIFIED
> Wrong injection made by container
> ---------------------------------
>
> Key: WFLY-82
> URL: https://issues.jboss.org/browse/WFLY-82
> Project: WildFly
> Issue Type: Bug
> Components: EE, EJB, JPA / Hibernate
> Reporter: Luan Cestari
> Assignee: Stuart Douglas
> Fix For: 8.0.0.Alpha1
>
>
> Description of problem:
> I got an issue in a salesforce case [1] about an incorrect injection.
> [1] https://c.na7.visual.force.com/apex/Case_View?id=500A000000CRqDG&sfdc.ove...
> Version-Release number of selected component (if applicable):
> How reproducible:
> To reproduce the case is required to have two PersistentUnit (PU), a Abstract class with a setter method to inject one PU and others classes inheriting the Abstract class and overriding the setter method to inject a different PU. In the runtime, the container inject twice the PU, first the correct PU and then the PU described in super class.
> I talked with the client (who is a Red Hat consultant) and we made the attached project based on a very simple JBoss example to reproduce the issue which happens in EAP6. I put a lot of 'println' to be very clear (even printing stacktrace to show who is calling). The most important files are:
> -> hibernate4Test/src/main/java/org/jboss/as/quickstart/hibernate4/data/BaseDAO.java -- The abstract class with the default PU injection
> -> hibernate4Test/src/main/java/org/jboss/as/quickstart/hibernate4/data/MemberRepository.java -- BaseDAO subclass
> -> hibernate4Test/src/main/java/org/jboss/as/quickstart/hibernate4/data/SecondaryMemberRepository.java -- BaseDAO subclass
> -> hibernate4Test/src/main/java/org/jboss/as/quickstart/hibernate4/data/MemberListProducer.java -- The class which inject the MemberListProducer and SecondaryMemberRepository
> -> hibernate4Test/src/main/resources/META-INF/persistence.xml -- We changed the configuration here to each PU have a different entity mapped
> Steps to Reproduce:
>
> Actual results:
> The EM injection occurs twice, which the last one the wrong EM.
> Expected results:
> Just one injection occurs.
> Additional info:
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years
[JBoss JIRA] (WFLY-88) Recovery not fully triggered when distributed transaction falls down at prepare phase of 2PC
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/WFLY-88?page=com.atlassian.jira.plugin.sy... ]
RH Bugzilla Integration commented on WFLY-88:
---------------------------------------------
Paul Gier <pgier(a)redhat.com> changed the Status of [bug 952746|https://bugzilla.redhat.com/show_bug.cgi?id=952746] from POST to MODIFIED
> Recovery not fully triggered when distributed transaction falls down at prepare phase of 2PC
> --------------------------------------------------------------------------------------------
>
> Key: WFLY-88
> URL: https://issues.jboss.org/browse/WFLY-88
> Project: WildFly
> Issue Type: Bug
> Components: EJB, Remoting
> Reporter: Ivo Studensky
> Assignee: jaikiran pai
> Fix For: 8.0.0.Alpha1
>
> Attachments: logs_prepareHaltClient.tgz
>
>
> It looks like recovery process is not fully triggered on a distributed transaction when the transaction falls down at prepare phase of 2PC. In the new crash recovery tests over propagated transactions only one of two servers recovers from the crash, but the other keeps an unfinished tx in its tx log.
> It corresponds to prepareHaltClient and prepareHaltServer test methods of org.jboss.as.test.jbossts.crashrec.txpropagation.TxPropagationCrashRecoveryTestCase, see JBQA-2604 for general description of the new tests. The prepareHaltClient test crashes the server which initiated the transaction, while as the prepareHaltServer test crashes the second server.
> The tests are written against EAP6.x branch, so for reproducing this it is needed a built server from the 7.1 branch of AS7.
> Steps to reproduce.
> 1. git clone -b as7 git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-transactions.git
> 2. cd eap-tests-transactions
> 3. git checkout tx_propag_crashrec_tests
> 4a. mvn clean verify -Dtest=TxPropagationCrashRecoveryTestCase#prepareHaltClient -Djboss.dist=<path to jboss-as-7.1.3.Final-SNAPSHOT>
> or
> 4b. mvn clean verify -Dtest=TxPropagationCrashRecoveryTestCase#prepareHaltServer -Djboss.dist=<path to jboss-as-7.1.3.Final-SNAPSHOT>
> The logs of prepareHaltClient run attached to this jira.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years