[aerogear-dev] UPS Production worthiness

Tue Jul 8 08:06:22 EDT 2014

Hi Matthias/UPS Team,

I have put together a patch which shows the changes I had to make to get the requisite performance from UPS. Please find the same attached. The patch has my changes along with Eric’s changes merged on 0.10.4 branch.  I had to do some hacky changes to prevent all the installations from getting serialized to UI. That should ideally be handled by removing this requirement and  making corresponding changes to UI code.

I will learn GIT and put together a PR for these changes as soon as I can. 

Thanks,

Vivek

From: mwessendorf at gmail.com [mailto:mwessendorf at gmail.com] On Behalf Of Matthias Wessendorf
Sent: Wednesday, July 02, 2014 8:15 PM
To: vivek.pandey at pinelabs.com
Cc: AeroGear Developer Mailing List
Subject: Re: [aerogear-dev] UPS Production worthiness

Hello vivek!

thanks for the reply! Absolutely a PR is more than welcome! If you have interest and the bandwidth, we would really appreciate that !

@importer: I still think it's a nice thing to have :-)

-Matthias

On Wed, Jul 2, 2014 at 4:40 PM, Vivek Pandey <vivek.pandey at pinelabs.com> wrote:

Hello Matthias,

An ‘importer’ might help, but smaller changes in existing domain/service might help more.

I ran a stress test on the installation endpoint yesterday, after merging Eric’s changes on source code of 0.10.x branch. 

I saw that even after using the merged code, the response worsened continuously. 

Looking closing, I saw that GenericVariantServiceImpl. addInstallation causes initialization of the now “lazy” collection AbstractVariant.installations.

I changed the model a bit, by adding variantID to Installation, and made it the owner of the variant-installation relationship.

That changed the code in InstallationRegistrationEndpoint. registerInstallation from

// store the installation:

            entity = clientInstallationService.addInstallation(entity);

            // add installation to the matching variant

            genericVariantService.addInstallation(variant, entity);

to 

                                                // store the installation:

                                                entity.setVariantID(variant);

                                                entity = clientInstallationService.addInstallation(entity);

This fixed the problem for me and now I am getting constant time installation registration with dramatically better performance

Please let me know if you want me to add this to AGPUSH-661

Thanks

Vivek

From: mwessendorf at gmail.com [mailto:mwessendorf at gmail.com] On Behalf Of Matthias Wessendorf
Sent: Wednesday, July 02, 2014 6:26 PM
To: AeroGear Developer Mailing List; vivek.pandey at pinelabs.com

Subject: Re: [aerogear-dev] UPS Production worthiness

Hello Vivek,

On Mon, May 26, 2014 at 1:23 PM, Matthias Wessendorf <matzew at apache.org> wrote:

On Mon, May 26, 2014 at 12:59 PM, Erik Jan de Wit <edewit at redhat.com> wrote:

Hi Vivek,

We’ve fixed that https://github.com/aerogear/aerogear-unifiedpush-server/commit/1357c9e834286a9227a90e9ab618226aa54bbbf3 the collection is now lazy. But I’m sure other query optimisations can be made.

+1 

btw. Vivek, I added your mail to an existing ticket:

https://issues.jboss.org/browse/AGPUSH-661

coming back from vacation and going over the JIRAs, I was wondering if an 'importer' helps?

That would be a RESTful endpoint that accepts a JSON file, containing all the devices you want to register (for a given variant).

That way it would be also a bit nicer on your side, to perform the import:

* generate a "large" JSON file (instead of a ton of requests)

* only one HTTP request

* server accepts and will process it in the background

I think that would be a nice migration tool.

Let me know what you think!

-Matthias

-Matthias

Cheers,

Erik Jan

On 26 May,2014, at 12:27 , Vivek Pandey <vivek.pandey at pinelabs.com> wrote:

Hi Matthias,

It is good to know that activity in java-apns is picking up and also that you are looking at pushy.

I did a few tests which added installations to UPS with a concurrency of 4-8 threads. I was using Postgres 9.3 and UPS 0.10.3 war

I noticed that response slowed down considerably after some time with high CPU usage and continued to get worse. After doing some profiling, I found that bulk of CPU cycles are being taken by org.postgresql.core.VisibleBufferedInputStream.readMore. The entire thread stack is attached. Also postgres continuously flagged

select installati0_.variantID as variant10_0_0_, installati0_.id as id1_3_0_, installati0_.id as id1_3_1_, installati0_.alias as alias2_3_1_, installati0_.deviceToken as deviceTo3_3_1_, installati0_.deviceType as deviceTy4_3_1_, installati0_.enabled as enabled5_3_1_, installati0_.operatingSystem as operatin6_3_1_, installati0_.osVersion as osVersio7_3_1_, installati0_.platform as platform8_3_1_, installati0_.simplePushEndpoint as simplePu9_3_1_ from InstallationImpl installati0_ where installati0_.variantID=$1

as the slow query. I am pretty sure that eager collection AbstractVariant.installations is the root cause of the problem.

Please let me know if you need any more information.

Thanks

Vivek

From:  <mailto:mwessendorf at gmail.com> mwessendorf at gmail.com [ <mailto:mwessendorf at gmail.com> mailto:mwessendorf at gmail.com] On Behalf Of Matthias Wessendorf
Sent: Wednesday, May 21, 2014 5:55 PM
To:  <mailto:vivek.pandey at pinelabs.com> vivek.pandey at pinelabs.com; AeroGear Developer Mailing List
Subject: Re: [aerogear-dev] UPS Production worthiness

Hello Vivek!

On Wed, May 21, 2014 at 2:07 PM, Vivek Pandey < <mailto:vivek.pandey at pinelabs.com> vivek.pandey at pinelabs.com> wrote:

Hi Jay,

Thanks for your reply.

While we have not faced any issues in using UPS in our limited testing, I often see info stacktraces in ups logs

2014-05-16 10:19:20,032 INFO  [com.notnoop.apns.internal.ApnsConnectionImpl] (Thread-118) Exception while waiting for error code: java.net.SocketException: Socket closed

        at java.net.SocketInputStream.socketRead0(Native Method) [rt.jar:1.7.0_51]

        at java.net.SocketInputStream.read(SocketInputStream.java:152) [rt.jar:1.7.0_51]

        at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51]

       ……………

        at com.notnoop.apns.internal.ApnsConnectionImpl$1MonitoringThread.run(ApnsConnectionImpl.java:114) [apns-0.2.3.jar:]

These stacktraces coupled with low dev activity of noop/java-apns project are disconcerting to me.

the stack-trace is no harm - it's only happening w/ doing a monitoring of the thread (that's what we currently do, when setting up ApnsService - I thought about explicitly disable that)

The activity of the underlying java-apns is very low, yes! However @froh42 is getting back:

 <https://github.com/notnoop/java-apns/commits/master> https://github.com/notnoop/java-apns/commits/master

There will be a new release in the near future; @froh42 asked me if I could help with pushing the bits to maven central

That said, I recently started looking at pushy:

 <https://github.com/relayrides/pushy> https://github.com/relayrides/pushy

I also sent a PR that would allow us to feed pushy w/ our certificate from the database:

 <https://github.com/relayrides/pushy/pull/87> https://github.com/relayrides/pushy/pull/87

Hope that helps

-Matthias

I am currently using UPS 0.10.2 war.

Thanks,

Vivek

From: Jay Balunas [mailto: <mailto:jbalunas at redhat.com> jbalunas at redhat.com] 
Sent: Tuesday, May 20, 2014 11:36 PM
To:  <mailto:vivek.pandey at pinelabs.com> vivek.pandey at pinelabs.com; AeroGear Developer Mailing List
Cc: Jay Balunas
Subject: Re: [aerogear-dev] UPS Production worthiness

Hi Vivek,

It's awesome to hear that you have integrated the UPS into your backend systems and some of your mobile apps!

We have a lot of confidence around the UPS, its functionality, and performance.  Our team has been working hard on improvements and stability including our QE team.  Also, as you may have seen we're planning a 1.0 release of the UPS this summer.

However at this time we don't have specific references or success stories outside of what you can see in the community mailing lists - other users using it ;-)  We're also about to kick off some performance and scale testing in the next couple of months.

Have you run into any issues that drove these questions about production worthiness?  If so please let us know and we'll certainly take a look.

Hope this helps!

Thanks,

Jay Balunas 

On May 20, 2014, at 8:09 AM, Vivek Pandey < <mailto:vivek.pandey at pinelabs.com> vivek.pandey at pinelabs.com> wrote:

Hello Aerogear dev team,

We integrated UPS into our backend server which is serving various mobile apps. While the development and testing phase went well, my manager is questioning me about production worthiness about Aerogear. It would be great help if you could point me to references/success stories where UPS is being used in production environments and scaling well in medium to high loads.

Thanks,

Vivek

  _____  

This message may contain privileged and confidential information and is solely for the use of intended recipient. The views expressed in this email are those of the sender and not of Pine Labs. The recipient should check this email and attachments for the presence of viruses / malwares etc. Pine Labs accepts no liability for any damage caused by any virus transmitted by this email. Pine Labs may monitor and record all emails.

  _____  

_______________________________________________
aerogear-dev mailing list
 <mailto:aerogear-dev at lists.jboss.org> aerogear-dev at lists.jboss.org
 <https://lists.jboss.org/mailman/listinfo/aerogear-dev> https://lists.jboss.org/mailman/listinfo/aerogear-dev

  _____  

This message may contain privileged and confidential information and is solely for the use of intended recipient. The views expressed in this email are those of the sender and not of Pine Labs. The recipient should check this email and attachments for the presence of viruses / malwares etc. Pine Labs accepts no liability for any damage caused by any virus transmitted by this email. Pine Labs may monitor and record all emails.

  _____  

_______________________________________________
aerogear-dev mailing list
 <mailto:aerogear-dev at lists.jboss.org> aerogear-dev at lists.jboss.org
 <https://lists.jboss.org/mailman/listinfo/aerogear-dev> https://lists.jboss.org/mailman/listinfo/aerogear-dev

-- 
Matthias Wessendorf 

blog:  <http://matthiaswessendorf.wordpress.com/> http://matthiaswessendorf.wordpress.com/
sessions:  <http://www.slideshare.net/mwessendorf> http://www.slideshare.net/mwessendorf
twitter:  <http://twitter.com/mwessendorf> http://twitter.com/mwessendorf

  _____  

This message may contain privileged and confidential information and is solely for the use of intended recipient. The views expressed in this email are those of the sender and not of Pine Labs. The recipient should check this email and attachments for the presence of viruses / malwares etc. Pine Labs accepts no liability for any damage caused by any virus transmitted by this email. Pine Labs may monitor and record all emails.

  _____  

<threadstack.txt>_______________________________________________

aerogear-dev mailing list
 <mailto:aerogear-dev at lists.jboss.org> aerogear-dev at lists.jboss.org
 <https://lists.jboss.org/mailman/listinfo/aerogear-dev> https://lists.jboss.org/mailman/listinfo/aerogear-dev

_______________________________________________
aerogear-dev mailing list
aerogear-dev at lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev

-- 
Matthias Wessendorf 

blog: http://matthiaswessendorf.wordpress.com/
sessions: http://www.slideshare.net/mwessendorf
twitter: http://twitter.com/mwessendorf 

-- 
Matthias Wessendorf 

blog: http://matthiaswessendorf.wordpress.com/
sessions: http://www.slideshare.net/mwessendorf
twitter: http://twitter.com/mwessendorf 

  _____  

This message may contain privileged and confidential information and is solely for the use of intended recipient. The views expressed in this email are those of the sender and not of Pine Labs. The recipient should check this email and attachments for the presence of viruses / malwares etc. Pine Labs accepts no liability for any damage caused by any virus transmitted by this email. Pine Labs may monitor and record all emails.

  _____  

-- 
Matthias Wessendorf 

blog: http://matthiaswessendorf.wordpress.com/
sessions: http://www.slideshare.net/mwessendorf
twitter: http://twitter.com/mwessendorf 

This message may contain privileged and confidential information and is solely for the use of intended recipient. The views expressed in this email are those of the sender and not of Pine Labs. The recipient should check this email and attachments for the presence of viruses / malwares etc. Pine Labs accepts no liability for any damage caused by any virus transmitted by this email. Pine Labs may monitor and record all emails.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/aerogear-dev/attachments/20140708/7eb325b5/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ups_changes_patch.patch
Type: application/octet-stream
Size: 13439 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/aerogear-dev/attachments/20140708/7eb325b5/attachment-0001.obj 