Re: [aerogear-dev] UPS Production worthiness

Hello Vivek! thanks a lot for the patch! We will take a look at that for sure! The benefit of getting a PR out is that you are actually captured inside of the GH as an official contributor :-) But thanks again, very happy to see your patch ! -Matthias On Tue, Jul 8, 2014 at 2:06 PM, Vivek Pandey <vivek.pandey(a)pinelabs.com&gt; wrote: > Hi Matthias/UPS Team, > > I have put together a patch which shows the changes I had to make to get > the requisite performance from UPS. Please find the same attached. The > patch has my changes along with Eric’s changes merged on 0.10.4 branch. I > had to do some hacky changes to prevent all the installations from getting > serialized to UI. That should ideally be handled by removing this > requirement and making corresponding changes to UI code. > > I will learn GIT and put together a PR for these changes as soon as I > can. > > > > Thanks, > > Vivek > > > > *From:* mwessendorf(a)gmail.com [mailto:mwessendorf@gmail.com] *On Behalf > Of *Matthias Wessendorf > *Sent:* Wednesday, July 02, 2014 8:15 PM > *To:* vivek.pandey(a)pinelabs.com > *Cc:* AeroGear Developer Mailing List > > *Subject:* Re: [aerogear-dev] UPS Production worthiness > > > > Hello vivek! > > > > thanks for the reply! Absolutely a PR is more than welcome! If you have > interest and the bandwidth, we would really appreciate that ! > > > > @importer: I still think it's a nice thing to have :-) > > > > -Matthias > > > > On Wed, Jul 2, 2014 at 4:40 PM, Vivek Pandey <vivek.pandey(a)pinelabs.com&gt; > wrote: > > Hello Matthias, > > > > An ‘importer’ might help, but smaller changes in existing domain/service > might help more. > > I ran a stress test on the installation endpoint yesterday, after merging > Eric’s changes on source code of 0.10.x branch. > > I saw that even after using the merged code, the response worsened > continuously. > > Looking closing, I saw that GenericVariantServiceImpl. addInstallation > causes initialization of the now “lazy” collection > AbstractVariant.installations. > > > > I changed the model a bit, by adding variantID to Installation, and made > it the owner of the variant-installation relationship. > > That changed the code in InstallationRegistrationEndpoint. registerInstallation > from > > // store the installation: > > entity = clientInstallationService.addInstallation(entity); > > // add installation to the matching variant > > genericVariantService.addInstallation(variant, entity); > > > > to > > // store the installation: > > > entity.setVariantID(variant); > > entity = > clientInstallationService.addInstallation(entity); > > > > This fixed the problem for me and now I am getting constant time > installation registration with dramatically better performance > > > > Please let me know if you want me to add this to AGPUSH-661 > > > > Thanks > > Vivek > > > > *From:* mwessendorf(a)gmail.com [mailto:mwessendorf@gmail.com] *On Behalf > Of *Matthias Wessendorf > *Sent:* Wednesday, July 02, 2014 6:26 PM > *To:* AeroGear Developer Mailing List; vivek.pandey(a)pinelabs.com > > > *Subject:* Re: [aerogear-dev] UPS Production worthiness > > > > Hello Vivek, > > > > On Mon, May 26, 2014 at 1:23 PM, Matthias Wessendorf <matzew(a)apache.org&gt; > wrote: > > > > > > On Mon, May 26, 2014 at 12:59 PM, Erik Jan de Wit <edewit(a)redhat.com&gt; > wrote: > > Hi Vivek, > > > > We’ve fixed that > https://github.com/aerogear/aerogear-unifiedpush-server/commit/1357c9e834... the > collection is now lazy. But I’m sure other query optimisations can be made. > > > > +1 > > > > btw. Vivek, I added your mail to an existing ticket: > > https://issues.jboss.org/browse/AGPUSH-661 > > > > > > coming back from vacation and going over the JIRAs, I was wondering if an > 'importer' helps? > > > > That would be a RESTful endpoint that accepts a JSON file, containing all > the devices you want to register (for a given variant). > > > > That way it would be also a bit nicer on your side, to perform the import: > > * generate a "large" JSON file (instead of a ton of requests) > > * only one HTTP request > > * server accepts and will process it in the background > > > > I think that would be a nice migration tool. > > > > Let me know what you think! > > > > -Matthias > > > > > > -Matthias > > > > > > Cheers, > > Erik Jan > > > > On 26 May,2014, at 12:27 , Vivek Pandey <vivek.pandey(a)pinelabs.com&gt; > wrote: > > > > Hi Matthias, > > > > It is good to know that activity in java-apns is picking up and also that > you are looking at pushy. > > > > I did a few tests which added installations to UPS with a concurrency of > 4-8 threads. I was using Postgres 9.3 and UPS 0.10.3 war > > > > I noticed that response slowed down considerably after some time with > high CPU usage and continued to get worse. After doing some profiling, I > found that bulk of CPU cycles are being taken by > org.postgresql.core.VisibleBufferedInputStream.readMore. The entire thread > stack is attached. Also postgres continuously flagged > > > > select installati0_.variantID as variant10_0_0_, installati0_.id as > id1_3_0_, installati0_.id as id1_3_1_, installati0_.alias as alias2_3_1_, > installati0_.deviceToken as deviceTo3_3_1_, installati0_.deviceType as > deviceTy4_3_1_, installati0_.enabled as enabled5_3_1_, > installati0_.operatingSystem as operatin6_3_1_, installati0_.osVersion as > osVersio7_3_1_, installati0_.platform as platform8_3_1_, > installati0_.simplePushEndpoint as simplePu9_3_1_ from InstallationImpl > installati0_ where installati0_.variantID=$1 > > > > as the slow query. I am pretty sure that eager collection > AbstractVariant.installations is the root cause of the problem. > > > > Please let me know if you need any more information. > > > > Thanks > > Vivek > > > > *From:* mwessendorf(a)gmail.com [mailto:mwessendorf@gmail.com > <mwessendorf(a)gmail.com>] *On Behalf Of *Matthias Wessendorf > *Sent:* Wednesday, May 21, 2014 5:55 PM > *To:* vivek.pandey(a)pinelabs.com; AeroGear Developer Mailing List > *Subject:* Re: [aerogear-dev] UPS Production worthiness > > > > Hello Vivek! > > > > On Wed, May 21, 2014 at 2:07 PM, Vivek Pandey <vivek.pandey(a)pinelabs.com&gt; > wrote: > > Hi Jay, > > > > Thanks for your reply. > > > > While we have not faced any issues in using UPS in our limited testing, I > often see info stacktraces in ups logs > > > > 2014-05-16 10:19:20,032 INFO > [com.notnoop.apns.internal.ApnsConnectionImpl] (Thread-118) Exception while > waiting for error code: java.net.SocketException: Socket closed > > at java.net.SocketInputStream.socketRead0(Native Method) > [rt.jar:1.7.0_51] > > at java.net.SocketInputStream.read(SocketInputStream.java:152) > [rt.jar:1.7.0_51] > > at java.net.SocketInputStream.read(SocketInputStream.java:122) > [rt.jar:1.7.0_51] > > …………… > > at > com.notnoop.apns.internal.ApnsConnectionImpl$1MonitoringThread.run(ApnsConnectionImpl.java:114) > [apns-0.2.3.jar:] > > > > > > These stacktraces coupled with low dev activity of noop/java-apns project > are disconcerting to me. > > > > the stack-trace is no harm - it's only happening w/ doing a monitoring of > the thread (that's what we currently do, when setting up ApnsService - I > thought about explicitly disable that) > > > > The activity of the underlying java-apns is very low, yes! However > @froh42 is getting back: > > https://github.com/notnoop/java-apns/commits/master > > > > There will be a new release in the near future; @froh42 asked me if I > could help with pushing the bits to maven central > > > > > > That said, I recently started looking at pushy: > > https://github.com/relayrides/pushy > > > > I also sent a PR that would allow us to feed pushy w/ our certificate > from the database: > > https://github.com/relayrides/pushy/pull/87 > > > > Hope that helps > > -Matthias > > > > > > I am currently using UPS 0.10.2 war. > > > > Thanks, > > Vivek > > *From:* Jay Balunas [mailto:jbalunas@redhat.com] > *Sent:* Tuesday, May 20, 2014 11:36 PM > *To:* vivek.pandey(a)pinelabs.com; AeroGear Developer Mailing List > *Cc:* Jay Balunas > *Subject:* Re: [aerogear-dev] UPS Production worthiness > > > > Hi Vivek, > > > > It's awesome to hear that you have integrated the UPS into your backend > systems and some of your mobile apps! > > > > We have a lot of confidence around the UPS, its functionality, and > performance. Our team has been working hard on improvements and stability > including our QE team. Also, as you may have seen we're planning a 1.0 > release of the UPS this summer. > > > > However at this time we don't have specific references or success stories > outside of what you can see in the community mailing lists - other users > using it ;-) We're also about to kick off some performance and scale > testing in the next couple of months. > > > > Have you run into any issues that drove these questions about production > worthiness? If so please let us know and we'll certainly take a look. > > > > Hope this helps! > > > > Thanks, > > Jay Balunas > > > > On May 20, 2014, at 8:09 AM, Vivek Pandey <vivek.pandey(a)pinelabs.com&gt; > wrote: > > > > Hello Aerogear dev team, > > > > We integrated UPS into our backend server which is serving various mobile > apps. While the development and testing phase went well, my manager is > questioning me about production worthiness about Aerogear. It would be > great help if you could point me to references/success stories where UPS is > being used in production environments and scaling well in medium to high > loads. > > > > Thanks, > > Vivek > > > > > > > ------------------------------ > > This message may contain privileged and confidential information and is > solely for the use of intended recipient. The views expressed in this email > are those of the sender and not of Pine Labs. The recipient should check > this email and attachments for the presence of viruses / malwares etc. Pine > Labs accepts no liability for any damage caused by any virus transmitted by > this email. Pine Labs may monitor and record all emails. > ------------------------------ > > > _______________________________________________ > aerogear-dev mailing list > aerogear-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/aerogear-dev > > > > > ------------------------------ > > This message may contain privileged and confidential information and is > solely for the use of intended recipient. The views expressed in this email > are those of the sender and not of Pine Labs. The recipient should check > this email and attachments for the presence of viruses / malwares etc. Pine > Labs accepts no liability for any damage caused by any virus transmitted by > this email. Pine Labs may monitor and record all emails. > ------------------------------ > > > > > _______________________________________________ > aerogear-dev mailing list > aerogear-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/aerogear-dev > > > > > > -- > Matthias Wessendorf > > blog: http://matthiaswessendorf.wordpress.com/ > sessions: http://www.slideshare.net/mwessendorf > twitter: http://twitter.com/mwessendorf > > > ------------------------------ > > This message may contain privileged and confidential information and is > solely for the use of intended recipient. The views expressed in this email > are those of the sender and not of Pine Labs. The recipient should check > this email and attachments for the presence of viruses / malwares etc. Pine > Labs accepts no liability for any damage caused by any virus transmitted by > this email. Pine Labs may monitor and record all emails. > ------------------------------ > > > > <threadstack.txt>_______________________________________________ > > > aerogear-dev mailing list > aerogear-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/aerogear-dev > > > > > _______________________________________________ > aerogear-dev mailing list > aerogear-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/aerogear-dev > > > > > > -- > Matthias Wessendorf > > blog: http://matthiaswessendorf.wordpress.com/ > sessions: http://www.slideshare.net/mwessendorf > twitter: http://twitter.com/mwessendorf > > > > > > -- > Matthias Wessendorf > > blog: http://matthiaswessendorf.wordpress.com/ > sessions: http://www.slideshare.net/mwessendorf > twitter: http://twitter.com/mwessendorf > > > ------------------------------ > > This message may contain privileged and confidential information and is > solely for the use of intended recipient. The views expressed in this email > are those of the sender and not of Pine Labs. The recipient should check > this email and attachments for the presence of viruses / malwares etc. Pine > Labs accepts no liability for any damage caused by any virus transmitted by > this email. Pine Labs may monitor and record all emails. > ------------------------------ > > > > > > > > -- > Matthias Wessendorf > > blog: http://matthiaswessendorf.wordpress.com/ > sessions: http://www.slideshare.net/mwessendorf > twitter: http://twitter.com/mwessendorf > > ------------------------------ > This message may contain privileged and confidential information and is > solely for the use of intended recipient. The views expressed in this email > are those of the sender and not of Pine Labs. The recipient should check > this email and attachments for the presence of viruses / malwares etc. Pine > Labs accepts no liability for any damage caused by any virus transmitted by > this email. Pine Labs may monitor and record all emails. > ------------------------------ > > -- Matthias Wessendorf blog: http://matthiaswessendorf.wordpress.com/ sessions: http://www.slideshare.net/mwessendorf twitter: http://twitter.com/mwessendorf