[aerogear-dev] Notification Delivery metrics and processing with Kafka

Wed May 17 05:58:19 EDT 2017

we have nothing yet - this is just getting started.

Over the summer we have two students, as part of the Google Summer of Code,
working on this.

I just did a POC to evaluate the idea ;-)

On Wed, May 17, 2017 at 7:27 AM, Mads Møller <mm at napp.dk> wrote:

> Are there Any guide/turorial on setting this up?
>
>
> MED VENLIG HILSEN / BEST REGARDS
> __________________
> MADS MØLLER
> CTO, PARTNER
>
> Napp A/S
> T: 42 42 80 60
> M: 20 28 20 26
> E: mm at napp.dk
> W: www.napp.dk
> __________________
>
> On 16 May 2017, at 21.46, Matthias Wessendorf <matzew at apache.org> wrote:
>
> right!
>
> in theory, you can also simply register a custom kafka consumer to those
> topics and apply some custom logic, e.g. to ping other systems
>
> On Tue, May 16, 2017 at 6:50 PM, Mads Møller <mm at napp.dk> wrote:
>
>> Such great news.
>>
>>
>>
>> It would be fantastic if failing APN connection or similar would fire off
>> a webhook. So its possible for integrated systems can be notified.
>>
>>
>>
>>
>>
>>
>>
>> BEST REGARDS
>>
>> *__________________*
>>
>> *MADS MØLLER*
>>
>> CTO, PARTNER
>>
>>
>>
>> Napp A/S
>>
>> T: +45 42 42 80 60 <+45%2042%2042%2080%2060>
>>
>> M: +45 20 28 20 26 <+45%2020%2028%2020%2026>
>>
>> E: mm at napp.dk
>>
>> W: https://napp.dk
>>
>> *__________________*
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *From: *<aerogear-dev-bounces at lists.jboss.org> on behalf of Matthias
>> Wessendorf <matzew at apache.org>
>> *Reply-To: *AeroGear Developer Mailing List <aerogear-dev at lists.jboss.org
>> >
>> *Date: *Tuesday, 16 May 2017 at 16.50
>> *To: *AeroGear Developer Mailing List <aerogear-dev at lists.jboss.org>
>> *Subject: *Re: [aerogear-dev] Notification Delivery metrics and
>> processing with Kafka
>>
>>
>>
>> Oh, this is far from being done :-)
>>
>>
>>
>> I just did a little POC, and since we also have two GSoC students, we
>> have some time to define the behavior, including UI, here on the commiunity
>> :)
>>
>>
>>
>> On Tue, May 16, 2017 at 3:37 PM, Jose Miguel Gallas Olmedo <
>> jgallaso at redhat.com> wrote:
>>
>> Great news!!
>>
>>
>>
>> Will this information be displayed in the UI? As a tooltip or when
>> extending the row in "activity log"s table.
>>
>>
>>
>> On 16 May 2017 at 13:58, Matthias Wessendorf <matzew at apache.org> wrote:
>>
>> Hi,
>>
>> with the new APNs HTTP/2 APIs, and our usage of Pushy, we are able to get
>> a way more finegrain knowledge if Apple did accept (for further processing)
>> or reject a messages, on a per device_token level!
>>
>> For instance, if we have a push with 5000 targeted devices, we are now
>> able to say that 5 tokens, for instances failed, but APNs was happy to
>> accept push request for the other 4995 devices (Note: this does NOT mean
>> they actually arrive at the device, just that apple accepted them for
>> further processing).
>>
>> Now, this, for APNs, gives us much more flexiblity handling our metrics!
>>
>> In our code, here, we do read *each* token request from APNs in here:
>> https://github.com/aerogear/aerogear-unifiedpush-serve
>> r/blob/20831d96196663349c96da6b5fe11aef65cacf59/push/sender/
>> src/main/java/org/jboss/aerogear/unifiedpush/message/
>> sender/apns/PushyApnsSender.java#L130-L147
>>
>> So here, we could simply send the result, on a per token base, to a
>> (Kafka) topic, like:
>>
>> ...
>>
>> if (pushNotificationResponse.isAccepted()) {
>>
>>   logger.trace("Push notification for '{}' (payload={})", deviceToken, pushNotificationResponse.getPushNotification().getPayload());
>>
>>
>>
>>   producer.send(jobID, "Success"); // sends to "push_messages" topic
>>
>> } else {
>>
>>   final String rejectReason = pushNotificationResponse.getRejectionReason();
>>
>>   logger.trace("Push Message has been rejected with reason: {}", rejectReason);
>>
>>
>>
>>   producer.send(jobID, "Rejected"); // sends "push_messages" topic
>>
>> ...
>>
>> }
>>
>> Now, this sends all to one topic, and we could be using, somewhere, Kafka
>> Stream API, to perform some processing of the source, and calculate some
>> stats on that, like:
>>
>> KStreamBuilder builder = new KStreamBuilder();
>>
>>
>>
>> // read from the topic that contains all messages, for all jobs
>>
>> final KStream<String, String> source = builder.stream("push_messages");
>>
>>
>>
>>
>>
>> // some simple processing, and grouping by key, applying a predicate and send to three "analytic" topic:
>>
>>
>>
>> final KTable<String, Long> successCountsPerJob = source.filter((key, value) -> value.equals("Success"))
>>
>>   .groupByKey()
>>
>>   .count("successMessagesPerJob");
>>
>> successCountsPerJob.to(Serdes.String(), Serdes.Long(), "successMessagesPerJob");
>>
>>
>>
>> final KTable<String, Long> failCountsPerJob = source.filter((key, value) -> value.equals("Rejected"))
>>
>>   .groupByKey()
>>
>>   .count("failedMessagesPerJob");
>>
>> failCountsPerJob.to(Serdes.String(), Serdes.Long(), "failedMessagesPerJob");
>>
>>
>>
>> source.groupByKey()
>>
>>   count("totalMessagesPerJob")
>>
>>     .to(Serdes.String(), Serdes.Long(), "totalMessagesPerJob");
>>
>>
>>
>> The above performs some functional processing of the single source of
>> truth, based on different assumptions.
>>
>> If one would have a simple consumer on each of these three "analytic"
>> topics, a simple logging output would be:
>>
>> 2017-05-16 13:42:48,763 INFO  successMessagesPerJob: 2 - jobID: XXX
>>
>> 2017-05-16 13:42:48,764 INFO  totalMessagesPerJob: 3 - jobID: XXX
>>
>> 2017-05-16 13:42:48,764 INFO  failedMessagesPerJob: 1 - jobID: XXX
>>
>>
>>
>> since for the GSoC we do have two students, working on Kafka and HBase
>> improvements for UPS, I wanted to share this quick prototype, as food for
>> thoughts.
>>
>> Of course, each of these 'filtered' consumers could than eventually store
>> the result somewhere else.
>>
>> With this approach, Kafka would be come the hub (or data pipeline) for
>> our metrics, with stream processing and different consumers to deal with
>> the results of interest
>>
>> Any comments or other thoughts?
>>
>> -Matthias
>>
>>
>>
>> --
>>
>> Matthias Wessendorf
>>
>> blog: http://matthiaswessendorf.wordpress.com/
>> twitter: http://twitter.com/mwessendorf
>>
>>
>>
>> _______________________________________________
>> aerogear-dev mailing list
>> aerogear-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>
>>
>>
>>
>>
>> --
>>
>> *JOSE MIGUEL GALLAS OLMEDO*
>>
>> ASSOCIATE QE, mobile
>>
>> Red Hat
>>
>> <https://www.redhat.com/>
>>
>> M: +34618488633 <http://redhatemailsignature-marketing.itos.redhat.com/>
>>
>>
>> <https://red.ht/sig>
>>
>>
>> _______________________________________________
>> aerogear-dev mailing list
>> aerogear-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>
>>
>>
>>
>>
>> --
>>
>> Matthias Wessendorf
>>
>> blog: http://matthiaswessendorf.wordpress.com/
>> twitter: http://twitter.com/mwessendorf
>>
>> _______________________________________________
>> aerogear-dev mailing list
>> aerogear-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>
>
>
>
> --
> Matthias Wessendorf
>
> blog: http://matthiaswessendorf.wordpress.com/
> twitter: http://twitter.com/mwessendorf
>
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
>
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>

-- 
Matthias Wessendorf

blog: http://matthiaswessendorf.wordpress.com/
twitter: http://twitter.com/mwessendorf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/aerogear-dev/attachments/20170517/60bff664/attachment-0001.html 