[JBoss JIRA] (JGRP-2135) OOM with JGroups 3.6.11.
by Bela Ban (Jira)
[ https://issues.redhat.com/browse/JGRP-2135?page=com.atlassian.jira.plugin... ]
Bela Ban commented on JGRP-2135:
--------------------------------
[~dereed]: the commit is {{c10f2be503167dded00964cf33706e20b85aa0b4}}. Will you create a one-off?
> OOM with JGroups 3.6.11.
> ------------------------
>
> Key: JGRP-2135
> URL: https://issues.redhat.com/browse/JGRP-2135
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.11
> Reporter: Zoltan Farkas
> Assignee: Bela Ban
> Priority: Major
> Fix For: 3.6.12
>
>
> We are running our JVMs with : -XX:OnOutOfMemoryError="kill -9 %p"
> we have been experiencing OOMs fairly often, and the OOMs happen at:
> {code}
> Object / Stack Frame |Name | Shallow Heap | Retained Heap |Context Class Loader |Is Daemon
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> java.lang.Thread @ 0x81bdf838 |Connection.Receiver [144.77.77.53:50363 - 144.77.77.53:50363],sis-cluster.service,prodpmwsv5-6461| 120 | 456 |sun.misc.Launcher$AppClassLoader @ 0x800175a8|false
> |- at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48) | | | | |
> |- at org.jgroups.blocks.cs.TcpConnection$Receiver.run()V (TcpConnection.java:310)| | | | |
> |- at java.lang.Thread.run()V (Thread.java:745) | | | | |
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> {code}
> the Code where it happens is in TcpConnection.java:
> {code}
> while(canRun()) {
> try {
> int len=in.readInt();
> if(buffer == null || buffer.length < len)
> buffer=new byte[len];
> in.readFully(buffer, 0, len);
> updateLastAccessed();
> server.receive(peer_addr, buffer, 0, len);
> }
> catch(OutOfMemoryError mem_ex) {
> t=mem_ex;
> break; // continue;
> }
> catch(IOException io_ex) {
> t=io_ex;
> break;
> }
> catch(Throwable e) {
> }
> }
> {code}
> when allocating: buffer=new byte[len];
> it looks to me that some invalid large value is received and the process OOMs when allocating a huge byte array
> Running JVMs without kill on OOM would make this issue "dissapear" in the sense that it is swallowed by:
> {code}
> catch(OutOfMemoryError mem_ex) {
> t=mem_ex;
> break; // continue;
> }
> {code}
> Handling OutOfMemoryError is a strange implementation choice...
> instead a size limit should be employed to protect from receiving invalid sizes...
> My heap limit is 1GB and my heap dumps are 50Mb so the attempted allocation size is huge...
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (JGRP-2135) OOM with JGroups 3.6.11.
by Bela Ban (Jira)
[ https://issues.redhat.com/browse/JGRP-2135?page=com.atlassian.jira.plugin... ]
Bela Ban commented on JGRP-2135:
--------------------------------
OK, I fixed this.
The root cause was *not* a spurious connection by a non-JGroups process, but the catching of {{Throwable}} in {{TcpConnection.run()}}: when there was an exception, the connection would not be closed, but we'd restart at the top of the loop, trying to read a new message. (Note that this could for example happen when the peer thread was interrupted trying to send a message, without closing its end of the connection).
However, when there was still stale data in the TCP pipe, we'd read the first 4 bytes and interpret them as length.
The reason the root cause was most likely not another non-JGroups process is that a non-JGroups process would have to send the correct cookie, version, and peer address at connection establishment time. Doable by a malicious process, but highly unlikely during regular processing.
The fix is that the loop in {{TcpConnection.run()}} now terminates on an exception, so that the next time we're sending data to the peer (or receive data from the peer), a new connection will need to be created.
This also implies we don't need a special {{max_size}} attribute.
> OOM with JGroups 3.6.11.
> ------------------------
>
> Key: JGRP-2135
> URL: https://issues.redhat.com/browse/JGRP-2135
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.11
> Reporter: Zoltan Farkas
> Assignee: Bela Ban
> Priority: Major
> Fix For: 3.6.12
>
>
> We are running our JVMs with : -XX:OnOutOfMemoryError="kill -9 %p"
> we have been experiencing OOMs fairly often, and the OOMs happen at:
> {code}
> Object / Stack Frame |Name | Shallow Heap | Retained Heap |Context Class Loader |Is Daemon
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> java.lang.Thread @ 0x81bdf838 |Connection.Receiver [144.77.77.53:50363 - 144.77.77.53:50363],sis-cluster.service,prodpmwsv5-6461| 120 | 456 |sun.misc.Launcher$AppClassLoader @ 0x800175a8|false
> |- at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48) | | | | |
> |- at org.jgroups.blocks.cs.TcpConnection$Receiver.run()V (TcpConnection.java:310)| | | | |
> |- at java.lang.Thread.run()V (Thread.java:745) | | | | |
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> {code}
> the Code where it happens is in TcpConnection.java:
> {code}
> while(canRun()) {
> try {
> int len=in.readInt();
> if(buffer == null || buffer.length < len)
> buffer=new byte[len];
> in.readFully(buffer, 0, len);
> updateLastAccessed();
> server.receive(peer_addr, buffer, 0, len);
> }
> catch(OutOfMemoryError mem_ex) {
> t=mem_ex;
> break; // continue;
> }
> catch(IOException io_ex) {
> t=io_ex;
> break;
> }
> catch(Throwable e) {
> }
> }
> {code}
> when allocating: buffer=new byte[len];
> it looks to me that some invalid large value is received and the process OOMs when allocating a huge byte array
> Running JVMs without kill on OOM would make this issue "dissapear" in the sense that it is swallowed by:
> {code}
> catch(OutOfMemoryError mem_ex) {
> t=mem_ex;
> break; // continue;
> }
> {code}
> Handling OutOfMemoryError is a strange implementation choice...
> instead a size limit should be employed to protect from receiving invalid sizes...
> My heap limit is 1GB and my heap dumps are 50Mb so the attempted allocation size is huge...
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (DROOLS-5280) List Filter expression returning single item and nested within arithmetic expression fails to evaluate
by Matteo Mortari (Jira)
[ https://issues.redhat.com/browse/DROOLS-5280?page=com.atlassian.jira.plug... ]
Matteo Mortari reassigned DROOLS-5280:
--------------------------------------
Assignee: Matteo Mortari (was: Mario Fusco)
> List Filter expression returning single item and nested within arithmetic expression fails to evaluate
> ------------------------------------------------------------------------------------------------------
>
> Key: DROOLS-5280
> URL: https://issues.redhat.com/browse/DROOLS-5280
> Project: Drools
> Issue Type: Bug
> Components: dmn engine
> Affects Versions: 7.34.0.Final, 7.36.0.Final
> Reporter: Tracy Hires
> Assignee: Matteo Mortari
> Priority: Major
> Attachments: MySpace_Relation.zip
>
>
> the expression `addend + listOfStructure[field1="some value"].numberField`, where `field1="some value"` filters `listOfStructure` to a single item and `addend` and `numberField` are both non-null numbers, fails to evaluate. It's possible to work around this by placing `listOfStructure[field1="some value"]` into a separate decision.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (WFLY-13358) Tests for WFCORE-4950 - Regression: Legacy Ldap Realm securing EJB with JDK8 not working
by Ricardo Martin Camarero (Jira)
[ https://issues.redhat.com/browse/WFLY-13358?page=com.atlassian.jira.plugi... ]
Ricardo Martin Camarero updated WFLY-13358:
-------------------------------------------
Summary: Tests for WFCORE-4950 - Regression: Legacy Ldap Realm securing EJB with JDK8 not working (was: Regression: Legacy Ldap Realm securing EJB with JDK8 not working)
> Tests for WFCORE-4950 - Regression: Legacy Ldap Realm securing EJB with JDK8 not working
> ----------------------------------------------------------------------------------------
>
> Key: WFLY-13358
> URL: https://issues.redhat.com/browse/WFLY-13358
> Project: WildFly
> Issue Type: Bug
> Components: EJB, Security
> Reporter: Darran Lofthouse
> Assignee: Ricardo Martin Camarero
> Priority: Critical
> Labels: downstream_dependency
> Fix For: 19.1.0.Final, 20.0.0.Beta1
>
>
> This is likely to actually be a WFCORE issue but as this is fixed a test case to reproduce should be added to the testsuite if appropriate.
> So far this seems a fairly generic issue an since WildFly 17 we must have run the testsuite against Java 8 100s if not 1000s of times so it is not appropriate to be reliant on an internal testsuite to detect the issue nine months or more later.
> The following issue describes the issue in relation to EAP: -
> https://issues.redhat.com/browse/JBEAP-19195
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (WFCORE-4950) Regression: Legacy Ldap Realm securing EJB with JDK8 not working
by Ricardo Martin Camarero (Jira)
Ricardo Martin Camarero created WFCORE-4950:
-----------------------------------------------
Summary: Regression: Legacy Ldap Realm securing EJB with JDK8 not working
Key: WFCORE-4950
URL: https://issues.redhat.com/browse/WFCORE-4950
Project: WildFly Core
Issue Type: Bug
Components: Security
Affects Versions: 12.0.0.Beta1
Reporter: Ricardo Martin Camarero
Assignee: Ricardo Martin Camarero
WFCORE issue related to JBEAP-19195. The root exception is the following:
{noformat}
javax.naming.NamingException: WFLYNAM0027: Failed instantiate InitialContextFactory com.sun.jndi.ldap.LdapCtxFactory from classloader ModuleClassLoader for Module "org.wildfly.extension.io" version 12.0.0.Beta1 from local module loader @5f2108b5 (finder: local module finder @31a5c39e (roots: /home/rmartinc/wildfly-20.0.0.Beta1-SNAPSHOT/modules,/home/rmartinc/wildfly-20.0.0.Beta1-SNAPSHOT/modules/system/layers/base)) [Root exception is java.lang.ClassNotFoundException: com.sun.jndi.ldap.LdapCtxFactory from [Module "org.wildfly.extension.io" version 12.0.0.Beta1 from local module loader @5f2108b5 (finder: local module finder @31a5c39e (roots: /home/rmartinc/wildfly-20.0.0.Beta1-SNAPSHOT/modules,/home/rmartinc/wildfly-20.0.0.Beta1-SNAPSHOT/modules/system/layers/base))]]
at org.jboss.as.naming.InitialContext.getDefaultInitCtx(InitialContext.java:120)
at org.jboss.as.naming.InitialContext.init(InitialContext.java:101)
at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154)
at org.jboss.as.naming.InitialContext.<init>(InitialContext.java:91)
at org.jboss.as.naming.InitialContextFactory.getInitialContext(InitialContextFactory.java:43)
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313)
at javax.naming.InitialContext.init(InitialContext.java:244)
at javax.naming.InitialContext.<init>(InitialContext.java:216)
at javax.naming.directory.InitialDirContext.<init>(InitialDirContext.java:101)
at org.jboss.as.domain.management.connections.ldap.LdapConnectionManagerService.getConnection(LdapConnectionManagerService.java:272)
at org.jboss.as.domain.management.connections.ldap.LdapConnectionManagerService.getConnection(LdapConnectionManagerService.java:184)
at org.jboss.as.domain.management.connections.ldap.LdapConnectionManagerService.getConnection(LdapConnectionManagerService.java:180)
at org.jboss.as.domain.management.security.LdapConnectionHandler.getConnection(LdapConnectionHandler.java:78)
at org.jboss.as.domain.management.security.LdapUserSearcherFactory$LdapUserSearcherImpl.search(LdapUserSearcherFactory.java:125)
at org.jboss.as.domain.management.security.LdapUserSearcherFactory$LdapUserSearcherImpl.search(LdapUserSearcherFactory.java:66)
at org.jboss.as.domain.management.security.LdapCacheService$NoCacheCache.search(LdapCacheService.java:232)
at org.jboss.as.domain.management.security.UserLdapCallbackHandler$SecurityRealmImpl.getRealmIdentity(UserLdapCallbackHandler.java:339)
at org.jboss.as.domain.management.security.SecurityRealmService$SharedStateSecurityRealm.getRealmIdentity(SecurityRealmService.java:776)
at org.wildfly.security.auth.server.ServerAuthenticationContext.assignName(ServerAuthenticationContext.java:1197)
at org.wildfly.security.auth.server.ServerAuthenticationContext$UnassignedState.setPrincipal(ServerAuthenticationContext.java:1726)
at org.wildfly.security.auth.server.ServerAuthenticationContext.setAuthenticationPrincipal(ServerAuthenticationContext.java:410)
at org.wildfly.security.auth.server.ServerAuthenticationContext.setAuthenticationName(ServerAuthenticationContext.java:384)
at org.wildfly.security.auth.server.ServerAuthenticationContext.setAuthenticationName(ServerAuthenticationContext.java:368)
at org.wildfly.security.auth.server.ServerAuthenticationContext$1.handleOne(ServerAuthenticationContext.java:912)
at org.wildfly.security.auth.server.ServerAuthenticationContext$1.handle(ServerAuthenticationContext.java:853)
at org.wildfly.security.auth.callback.SocketAddressQueryCallbackHandler.handle(SocketAddressQueryCallbackHandler.java:57)
at org.wildfly.security.sasl.util.TrustManagerSaslServerFactory.lambda$createSaslServer$0(TrustManagerSaslServerFactory.java:105)
at org.wildfly.security.sasl.plain.PlainSaslServer.evaluateResponse(PlainSaslServer.java:118)
at org.wildfly.security.sasl.util.AuthenticationCompleteCallbackSaslServerFactory$1.evaluateResponse(AuthenticationCompleteCallbackSaslServerFactory.java:58)
at org.wildfly.security.sasl.util.AuthenticationTimeoutSaslServerFactory$DelegatingTimeoutSaslServer.evaluateResponse(AuthenticationTimeoutSaslServerFactory.java:110)
at org.wildfly.security.sasl.util.SecurityIdentitySaslServerFactory$1.evaluateResponse(SecurityIdentitySaslServerFactory.java:59)
at org.xnio.sasl.SaslUtils.evaluateResponse(SaslUtils.java:245)
at org.xnio.sasl.SaslUtils.evaluateResponse(SaslUtils.java:217)
at org.jboss.remoting3.remote.ServerConnectionOpenListener$AuthStepRunnable.run(ServerConnectionOpenListener.java:484)
at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:991)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.sun.jndi.ldap.LdapCtxFactory from [Module "org.wildfly.extension.io" version 12.0.0.Beta1 from local module loader @5f2108b5 (finder: local module finder @31a5c39e (roots: /home/rmartinc/wildfly-20.0.0.Beta1-SNAPSHOT/modules,/home/rmartinc/wildfly-20.0.0.Beta1-SNAPSHOT/modules/system/layers/base))]
at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:255)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.jboss.as.naming.InitialContext.getDefaultInitCtx(InitialContext.java:115)
... 40 more
{noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (WFLY-13358) Regression: Legacy Ldap Realm securing EJB with JDK8 not working
by Ricardo Martin Camarero (Jira)
[ https://issues.redhat.com/browse/WFLY-13358?page=com.atlassian.jira.plugi... ]
Ricardo Martin Camarero reassigned WFLY-13358:
----------------------------------------------
Assignee: Ricardo Martin Camarero
> Regression: Legacy Ldap Realm securing EJB with JDK8 not working
> ----------------------------------------------------------------
>
> Key: WFLY-13358
> URL: https://issues.redhat.com/browse/WFLY-13358
> Project: WildFly
> Issue Type: Bug
> Components: EJB, Security
> Reporter: Darran Lofthouse
> Assignee: Ricardo Martin Camarero
> Priority: Critical
> Labels: downstream_dependency
> Fix For: 19.1.0.Final, 20.0.0.Beta1
>
>
> This is likely to actually be a WFCORE issue but as this is fixed a test case to reproduce should be added to the testsuite if appropriate.
> So far this seems a fairly generic issue an since WildFly 17 we must have run the testsuite against Java 8 100s if not 1000s of times so it is not appropriate to be reliant on an internal testsuite to detect the issue nine months or more later.
> The following issue describes the issue in relation to EAP: -
> https://issues.redhat.com/browse/JBEAP-19195
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (WFLY-5097) @Startup @Singleton Beans Constructed Multiple Times
by Panagiotis Sotiropoulos (Jira)
[ https://issues.redhat.com/browse/WFLY-5097?page=com.atlassian.jira.plugin... ]
Panagiotis Sotiropoulos reassigned WFLY-5097:
---------------------------------------------
Assignee: Panagiotis Sotiropoulos
> @Startup @Singleton Beans Constructed Multiple Times
> ----------------------------------------------------
>
> Key: WFLY-5097
> URL: https://issues.redhat.com/browse/WFLY-5097
> Project: WildFly
> Issue Type: Bug
> Components: EJB
> Affects Versions: 9.0.1.Final
> Environment: All
> Reporter: Davide Crudo
> Assignee: Panagiotis Sotiropoulos
> Priority: Major
> Attachments: singleton.zip
>
>
> When using @Startup and/or @Singleton to start a Bean on server startup, everytime a method is accessed or the bean is injected with @EJB, both Constructor and @Postconstruct are executed.
> This happens multiple times, each time any component uses the Singleton bean.
> This seems similar to an issue reported for JBOSS AS 7.x (see forum link reference below)
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years