7:14 p.m.
Hi,
We use the OpenShift cartridge
<https://github.com/aerogear/openshift-origin-cartridge-aerogear-push> to
deploy AeroGear UnifiedPush server version 1.0.3. We deploy it on a medium
gear.
Am I expected to increase the maximum heap size for JVM, set by default to
be 512m?
We are getting OutOfMemoryError frequently. And it's often very easily
reproducible by sending 10 or so consecutive messages from AeroGear's admin
console.
I can provide more details, but I wanted to see if I was supposed to
increase the heap size.
Thank you.
--
View this message in context:
http://aerogear-users.1116366.n5.nabble.com/Default-heap-size-and-OOM-tp5...
Sent from the aerogear-users mailing list archive at Nabble.com.
10:30 a.m.
Hi,
Farah and I are looking to remove the JGroups subsystem. This has been a
boost on other cartridges (that do not need clustering) as well.
Will keep you updated
On Wed, Jun 17, 2015 at 7:14 PM, mo <michi.oshima(a)gmail.com> wrote:
Hi,
We use the OpenShift cartridge
<https://github.com/aerogear/openshift-origin-cartridge-aerogear-push>
to
deploy AeroGear UnifiedPush server version 1.0.3. We deploy it on a medium
gear.
Am I expected to increase the maximum heap size for JVM, set by default to
be 512m?
We are getting OutOfMemoryError frequently. And it's often very easily
reproducible by sending 10 or so consecutive messages from AeroGear's admin
console.
I can provide more details, but I wanted to see if I was supposed to
increase the heap size.
Thank you.
--
View this message in context:
http://aerogear-users.1116366.n5.nabble.com/Default-heap-size-and-OOM-tp5...
Sent from the aerogear-users mailing list archive at Nabble.com.
_______________________________________________
Aerogear-users mailing list
Aerogear-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-users
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
sessions: http://www.slideshare.net/mwessendorf
twitter: http://twitter.com/mwessendorf
4:57 p.m.
Thanks, Matthias. I take your reply to mean "No, one shouldn't have to
increase JVM heap size."
Let me list some more parameters, in the hope that you might spot something
obvious that we can correct on our side. It's been difficult to
troubleshoot this, because when the problem happens we can't even SSH into
the server. I believe whatever the problem is, it's exhausting resources on
the server.
Parameters:
1. Deployed on OpenShift using the cartridge found here:
https://github.com/aerogear/openshift-origin-cartridge-aerogear-push
<https://github.com/aerogear/openshift-origin-cartridge-aerogear-push>
2. AeroGear UnifiedPush Server 1.0.3 and WildFly, MySQL 5.5, Medium gear.
3. One application is defined with one variant (simple-push), 131
installations registered.
4. We are using all default settings. We haven't customized the JVM max
heap size, for example. The server log tells me "-Xmx512m".
Here are the reproduction steps. This happened reliably two days ago. We
couldn't reproduce this yesterday after a fresh restart of AeroGear.
1. Logged into AeroGear as admin
2. Select "Send Push"
3. Select an application
4. Hit "Send Push Notification", multiple times one after another. About 10
consecutive messages would cause OutOfMemoryError on our server.
Here's a relevant snippet of the server log:
[0m[0m17:03:29,511 INFO [PushNotificationSenderEndpoint] (default
task-28) Message submitted to PushNetworks for further processing
[0m[0m17:03:29,513 INFO [JPAPushMessageInformationDao] (EJB default - 6)
Processing send request with '[alert=, criteria=[aliases=null,
deviceTypes=null, categories=null, variants=null], time-to-live=-1]'
payload
[0m[31m17:03:29,837 ERROR [org.jboss.as.ejb3.invocation] (EJB default - 6)
JBAS014134: EJB Invocation failed on component SenderServiceImpl for method
public abstract void
org.jboss.aerogear.unifiedpush.message.SenderService.send(org.jboss.aerogear.unifiedpush.api.PushApplication,org.jboss.aerogear.unifiedpush.message.UnifiedPushMessage):
javax.ejb.EJBException: JBAS014580: Unexpected Error
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
.
.
.
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[rt.jar:1.7.0_79]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[rt.jar:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_79]
at java.lang.Thread.start(Thread.java:714) [rt.jar:1.7.0_79]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1056)
[jsse.jar:1.7.0_79]
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
[jsse.jar:1.7.0_79]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359)
[jsse.jar:1.7.0_79]
.
.
.
org.jboss.aerogear.unifiedpush.message.sender.SimplePushNotificationSender.put(SimplePushNotificationSender.java:111)
[unifiedpush-push-1.0.3.jar:1.0.3]
at
org.jboss.aerogear.unifiedpush.message.sender.SimplePushNotificationSender.sendPushMessage(SimplePushNotificationSender.java:65)
[unifiedpush-push-1.0.3.jar:1.0.3]
at
org.jboss.aerogear.unifiedpush.message.SenderServiceImpl.send(SenderServiceImpl.java:110)
[unifiedpush-push-1.0.3.jar:1.0.3]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.7.0_79]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[rt.jar:1.7.0_79]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
at
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
.
.
.
at
org.jboss.as.ejb3.component.interceptors.NonPooledEJBComponentInstanceAssociatingInterceptor.processInvocation(NonPooledEJBComponentInstanceAssociatingInterceptor.java:59)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:273)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
... 38 more
--
View this message in context:
http://aerogear-users.1116366.n5.nabble.com/Default-heap-size-and-OOM-tp5...
Sent from the aerogear-users mailing list archive at Nabble.com.
5:17 p.m.
Hi ,
[1]
https://github.com/aerogear/openshift-origin-cartridge-aerogear-push/pull/18
On Thu, Jun 18, 2015 at 4:57 PM, mo <michi.oshima(a)gmail.com> wrote:
Thanks, Matthias. I take your reply to mean "No, one
shouldn't have to
increase JVM heap size."
yeah, if you really want to, you can try tweaking the jvm heap size (as
long as you still use a function of the max memory available on the gear)
But also please review the changes done in
https://github.com/aerogear/openshift-origin-cartridge-aerogear-push/pull/18
You could port them over to your instance. Makes sense ?
More coming soon.
Let me list some more parameters, in the hope that you might spot something
obvious that we can correct on our side. It's been difficult to
troubleshoot this, because when the problem happens we can't even SSH into
the server. I believe whatever the problem is, it's exhausting resources
on
the server.
Parameters:
1. Deployed on OpenShift using the cartridge found here:
https://github.com/aerogear/openshift-origin-cartridge-aerogear-push
<https://github.com/aerogear/openshift-origin-cartridge-aerogear-push>
2. AeroGear UnifiedPush Server 1.0.3 and WildFly, MySQL 5.5, Medium gear.
3. One application is defined with one variant (simple-push), 131
installations registered.
4. We are using all default settings. We haven't customized the JVM max
heap size, for example. The server log tells me "-Xmx512m".
Here are the reproduction steps. This happened reliably two days ago. We
couldn't reproduce this yesterday after a fresh restart of AeroGear.
1. Logged into AeroGear as admin
2. Select "Send Push"
3. Select an application
4. Hit "Send Push Notification", multiple times one after another. About
10
consecutive messages would cause OutOfMemoryError on our server.
Here's a relevant snippet of the server log:
[0m [0m17:03:29,511 INFO [PushNotificationSenderEndpoint] (default
task-28) Message submitted to PushNetworks for further processing
[0m [0m17:03:29,513 INFO [JPAPushMessageInformationDao] (EJB default - 6)
Processing send request with '[alert=, criteria=[aliases=null,
deviceTypes=null, categories=null, variants=null], time-to-live=-1]'
payload
[0m [31m17:03:29,837 ERROR [org.jboss.as.ejb3.invocation] (EJB default -
6)
JBAS014134: EJB Invocation failed on component SenderServiceImpl for method
public abstract void
org.jboss.aerogear.unifiedpush.message.SenderService.send(org.jboss.aerogear.unifiedpush.api.PushApplication,org.jboss.aerogear.unifiedpush.message.UnifiedPushMessage):
javax.ejb.EJBException: JBAS014580: Unexpected Error
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
.
.
.
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[rt.jar:1.7.0_79]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[rt.jar:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_79]
at java.lang.Thread.start(Thread.java:714) [rt.jar:1.7.0_79]
at
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1056)
[jsse.jar:1.7.0_79]
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
[jsse.jar:1.7.0_79]
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359)
[jsse.jar:1.7.0_79]
.
.
.
org.jboss.aerogear.unifiedpush.message.sender.SimplePushNotificationSender.put(SimplePushNotificationSender.java:111)
[unifiedpush-push-1.0.3.jar:1.0.3]
at
org.jboss.aerogear.unifiedpush.message.sender.SimplePushNotificationSender.sendPushMessage(SimplePushNotificationSender.java:65)
[unifiedpush-push-1.0.3.jar:1.0.3]
at
org.jboss.aerogear.unifiedpush.message.SenderServiceImpl.send(SenderServiceImpl.java:110)
[unifiedpush-push-1.0.3.jar:1.0.3]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.7.0_79]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[rt.jar:1.7.0_79]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606)
[rt.jar:1.7.0_79]
at
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
.
.
.
at
org.jboss.as.ejb3.component.interceptors.NonPooledEJBComponentInstanceAssociatingInterceptor.processInvocation(NonPooledEJBComponentInstanceAssociatingInterceptor.java:59)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:273)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
... 38 more
--
View this message in context:
http://aerogear-users.1116366.n5.nabble.com/Default-heap-size-and-OOM-tp5...
Sent from the aerogear-users mailing list archive at Nabble.com.
_______________________________________________
Aerogear-users mailing list
Aerogear-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-users
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
sessions: http://www.slideshare.net/mwessendorf
twitter: http://twitter.com/mwessendorf
3:06 p.m.
Hi Mo,
the PR has been merged, and it should improve the situation a bit
PS: next will be removing SimplePush from the cartridge
On Thu, Jun 18, 2015 at 5:17 PM, Matthias Wessendorf <matzew(a)apache.org>
wrote:
Hi ,
[1]
https://github.com/aerogear/openshift-origin-cartridge-aerogear-push/pull/18
On Thu, Jun 18, 2015 at 4:57 PM, mo <michi.oshima(a)gmail.com> wrote:
> Thanks, Matthias. I take your reply to mean "No, one shouldn't have to
> increase JVM heap size."
>
yeah, if you really want to, you can try tweaking the jvm heap size (as
long as you still use a function of the max memory available on the gear)
But also please review the changes done in
https://github.com/aerogear/openshift-origin-cartridge-aerogear-push/pull/18
You could port them over to your instance. Makes sense ?
More coming soon.
>
> Let me list some more parameters, in the hope that you might spot
> something
> obvious that we can correct on our side. It's been difficult to
> troubleshoot this, because when the problem happens we can't even SSH into
> the server. I believe whatever the problem is, it's exhausting resources
> on
> the server.
>
> Parameters:
>
> 1. Deployed on OpenShift using the cartridge found here:
> https://github.com/aerogear/openshift-origin-cartridge-aerogear-push
> <https://github.com/aerogear/openshift-origin-cartridge-aerogear-push>
> 2. AeroGear UnifiedPush Server 1.0.3 and WildFly, MySQL 5.5, Medium gear.
> 3. One application is defined with one variant (simple-push), 131
> installations registered.
> 4. We are using all default settings. We haven't customized the JVM max
> heap size, for example. The server log tells me "-Xmx512m".
>
> Here are the reproduction steps. This happened reliably two days ago. We
> couldn't reproduce this yesterday after a fresh restart of AeroGear.
>
> 1. Logged into AeroGear as admin
> 2. Select "Send Push"
> 3. Select an application
> 4. Hit "Send Push Notification", multiple times one after another. About
> 10
> consecutive messages would cause OutOfMemoryError on our server.
>
> Here's a relevant snippet of the server log:
>
> [0m [0m17:03:29,511 INFO [PushNotificationSenderEndpoint] (default
> task-28) Message submitted to PushNetworks for further processing
> [0m [0m17:03:29,513 INFO [JPAPushMessageInformationDao] (EJB default -
> 6)
> Processing send request with '[alert=, criteria=[aliases=null,
> deviceTypes=null, categories=null, variants=null], time-to-live=-1]'
> payload
> [0m [31m17:03:29,837 ERROR [org.jboss.as.ejb3.invocation] (EJB default -
> 6)
> JBAS014134: EJB Invocation failed on component SenderServiceImpl for
> method
> public abstract void
>
>
org.jboss.aerogear.unifiedpush.message.SenderService.send(org.jboss.aerogear.unifiedpush.api.PushApplication,org.jboss.aerogear.unifiedpush.message.UnifiedPushMessage):
> javax.ejb.EJBException: JBAS014580: Unexpected Error
> at
>
>
org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> [wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
> at
>
> org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
> [wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
>
> .
> .
> .
>
> [wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [rt.jar:1.7.0_79]
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [rt.jar:1.7.0_79]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_79]
> at java.lang.Thread.start(Thread.java:714) [rt.jar:1.7.0_79]
> at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1056)
> [jsse.jar:1.7.0_79]
> at
>
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
> [jsse.jar:1.7.0_79]
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359)
> [jsse.jar:1.7.0_79]
> .
> .
> .
>
>
>
org.jboss.aerogear.unifiedpush.message.sender.SimplePushNotificationSender.put(SimplePushNotificationSender.java:111)
> [unifiedpush-push-1.0.3.jar:1.0.3]
> at
>
>
org.jboss.aerogear.unifiedpush.message.sender.SimplePushNotificationSender.sendPushMessage(SimplePushNotificationSender.java:65)
> [unifiedpush-push-1.0.3.jar:1.0.3]
> at
>
>
org.jboss.aerogear.unifiedpush.message.SenderServiceImpl.send(SenderServiceImpl.java:110)
> [unifiedpush-push-1.0.3.jar:1.0.3]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [rt.jar:1.7.0_79]
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [rt.jar:1.7.0_79]
> at
>
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [rt.jar:1.7.0_79]
> at java.lang.reflect.Method.invoke(Method.java:606)
> [rt.jar:1.7.0_79]
> at
>
>
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
>
> .
> .
> .
>
> at
>
>
org.jboss.as.ejb3.component.interceptors.NonPooledEJBComponentInstanceAssociatingInterceptor.processInvocation(NonPooledEJBComponentInstanceAssociatingInterceptor.java:59)
> [wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
> at
>
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
>
> org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:273)
> [wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
> ... 38 more
>
>
>
>
> --
> View this message in context:
> http://aerogear-users.1116366.n5.nabble.com/Default-heap-size-and-OOM-tp5...
> Sent from the aerogear-users mailing list archive at Nabble.com.
>
> _______________________________________________
> Aerogear-users mailing list
> Aerogear-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-users
>
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
sessions: http://www.slideshare.net/mwessendorf
twitter: http://twitter.com/mwessendorf
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
sessions: http://www.slideshare.net/mwessendorf
twitter: http://twitter.com/mwessendorf
4:09 p.m.
Awesome. Thank you, Matthias.
I'm currently revising our client side code, as it is likely contributing to
wasted resources on the server side. I will post a report after the
revision and testing.
--
View this message in context:
http://aerogear-users.1116366.n5.nabble.com/Default-heap-size-and-OOM-tp5...
Sent from the aerogear-users mailing list archive at Nabble.com.
3238
days inactive
3244
days old
aerogear-users@lists.jboss.org
5 comments
2 participants
participants (2)
-
Matthias Wessendorf -
mo