1:58 p.m.
Hi,
with the new APNs HTTP/2 APIs, and our usage of Pushy, we are able to get a
way more finegrain knowledge if Apple did accept (for further processing)
or reject a messages, on a per device_token level!
For instance, if we have a push with 5000 targeted devices, we are now able
to say that 5 tokens, for instances failed, but APNs was happy to accept
push request for the other 4995 devices (Note: this does NOT mean they
actually arrive at the device, just that apple accepted them for further
processing).
Now, this, for APNs, gives us much more flexiblity handling our metrics!
In our code, here, we do read *each* token request from APNs in here:
https://github.com/aerogear/aerogear-unifiedpush-server/blob/20831d961966...
So here, we could simply send the result, on a per token base, to a (Kafka)
topic, like:
...
if (pushNotificationResponse.isAccepted()) {
logger.trace("Push notification for '{}' (payload={})", deviceToken,
pushNotificationResponse.getPushNotification().getPayload());
producer.send(jobID, "Success"); // sends to "push_messages" topic
} else {
final String rejectReason = pushNotificationResponse.getRejectionReason();
logger.trace("Push Message has been rejected with reason: {}", rejectReason);
producer.send(jobID, "Rejected"); // sends "push_messages" topic
...
}
Now, this sends all to one topic, and we could be using, somewhere, Kafka
Stream API, to perform some processing of the source, and calculate some
stats on that, like:
KStreamBuilder builder = new KStreamBuilder();
// read from the topic that contains all messages, for all jobs
final KStream<String, String> source = builder.stream("push_messages");
// some simple processing, and grouping by key, applying a predicate
and send to three "analytic" topic:
final KTable<String, Long> successCountsPerJob = source.filter((key,
value) -> value.equals("Success"))
.groupByKey()
.count("successMessagesPerJob");
successCountsPerJob.to(Serdes.String(), Serdes.Long(),
"successMessagesPerJob");
final KTable<String, Long> failCountsPerJob = source.filter((key,
value) -> value.equals("Rejected"))
.groupByKey()
.count("failedMessagesPerJob");
failCountsPerJob.to(Serdes.String(), Serdes.Long(), "failedMessagesPerJob");
source.groupByKey()
count("totalMessagesPerJob")
.to(Serdes.String(), Serdes.Long(), "totalMessagesPerJob");
The above performs some functional processing of the single source of
truth, based on different assumptions.
If one would have a simple consumer on each of these three "analytic"
topics, a simple logging output would be:
2017-05-16 13:42:48,763 INFO successMessagesPerJob: 2 - jobID: XXX
2017-05-16 13:42:48,764 INFO totalMessagesPerJob: 3 - jobID: XXX
2017-05-16 13:42:48,764 INFO failedMessagesPerJob: 1 - jobID: XXX
since for the GSoC we do have two students, working on Kafka and HBase
improvements for UPS, I wanted to share this quick prototype, as food for
thoughts.
Of course, each of these 'filtered' consumers could than eventually store
the result somewhere else.
With this approach, Kafka would be come the hub (or data pipeline) for our
metrics, with stream processing and different consumers to deal with the
results of interest
Any comments or other thoughts?
-Matthias
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
twitter: http://twitter.com/mwessendorf
3:37 p.m.
Great news!!
Will this information be displayed in the UI? As a tooltip or when
extending the row in "activity log"s table.
On 16 May 2017 at 13:58, Matthias Wessendorf <matzew(a)apache.org> wrote:
Hi,
with the new APNs HTTP/2 APIs, and our usage of Pushy, we are able to get
a way more finegrain knowledge if Apple did accept (for further processing)
or reject a messages, on a per device_token level!
For instance, if we have a push with 5000 targeted devices, we are now
able to say that 5 tokens, for instances failed, but APNs was happy to
accept push request for the other 4995 devices (Note: this does NOT mean
they actually arrive at the device, just that apple accepted them for
further processing).
Now, this, for APNs, gives us much more flexiblity handling our metrics!
In our code, here, we do read *each* token request from APNs in here:
https://github.com/aerogear/aerogear-unifiedpush-server/blob/
20831d96196663349c96da6b5fe11aef65cacf59/push/sender/src/
main/java/org/jboss/aerogear/unifiedpush/message/sender/
apns/PushyApnsSender.java#L130-L147
So here, we could simply send the result, on a per token base, to a
(Kafka) topic, like:
...
if (pushNotificationResponse.isAccepted()) {
logger.trace("Push notification for '{}' (payload={})", deviceToken,
pushNotificationResponse.getPushNotification().getPayload());
producer.send(jobID, "Success"); // sends to "push_messages" topic
} else {
final String rejectReason = pushNotificationResponse.getRejectionReason();
logger.trace("Push Message has been rejected with reason: {}",
rejectReason);
producer.send(jobID, "Rejected"); // sends "push_messages" topic
...
}
Now, this sends all to one topic, and we could be using, somewhere, Kafka
Stream API, to perform some processing of the source, and calculate some
stats on that, like:
KStreamBuilder builder = new KStreamBuilder();
// read from the topic that contains all messages, for all jobs
final KStream<String, String> source = builder.stream("push_messages");
// some simple processing, and grouping by key, applying a predicate and send to three
"analytic" topic:
final KTable<String, Long> successCountsPerJob = source.filter((key, value) ->
value.equals("Success"))
.groupByKey()
.count("successMessagesPerJob");
successCountsPerJob.to(Serdes.String(), Serdes.Long(),
"successMessagesPerJob");
final KTable<String, Long> failCountsPerJob = source.filter((key, value) ->
value.equals("Rejected"))
.groupByKey()
.count("failedMessagesPerJob");
failCountsPerJob.to(Serdes.String(), Serdes.Long(), "failedMessagesPerJob");
source.groupByKey()
count("totalMessagesPerJob")
.to(Serdes.String(), Serdes.Long(), "totalMessagesPerJob");
The above performs some functional processing of the single source of
truth, based on different assumptions.
If one would have a simple consumer on each of these three "analytic"
topics, a simple logging output would be:
2017-05-16 13:42:48,763 INFO successMessagesPerJob: 2 - jobID: XXX
2017-05-16 13:42:48,764 INFO totalMessagesPerJob: 3 - jobID: XXX
2017-05-16 13:42:48,764 INFO failedMessagesPerJob: 1 - jobID: XXX
since for the GSoC we do have two students, working on Kafka and HBase
improvements for UPS, I wanted to share this quick prototype, as food for
thoughts.
Of course, each of these 'filtered' consumers could than eventually store
the result somewhere else.
With this approach, Kafka would be come the hub (or data pipeline) for our
metrics, with stream processing and different consumers to deal with the
results of interest
Any comments or other thoughts?
-Matthias
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
twitter: http://twitter.com/mwessendorf
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
--
JOSE MIGUEL GALLAS OLMEDO
ASSOCIATE QE, mobile
Red Hat
<https://www.redhat.com/>
M: +34618488633 <http://redhatemailsignature-marketing.itos.redhat.com/>
<https://red.ht/sig>
4:50 p.m.
Oh, this is far from being done :-)
I just did a little POC, and since we also have two GSoC students, we have
some time to define the behavior, including UI, here on the commiunity :)
On Tue, May 16, 2017 at 3:37 PM, Jose Miguel Gallas Olmedo <
jgallaso(a)redhat.com> wrote:
Great news!!
Will this information be displayed in the UI? As a tooltip or when
extending the row in "activity log"s table.
On 16 May 2017 at 13:58, Matthias Wessendorf <matzew(a)apache.org> wrote:
> Hi,
>
> with the new APNs HTTP/2 APIs, and our usage of Pushy, we are able to get
> a way more finegrain knowledge if Apple did accept (for further processing)
> or reject a messages, on a per device_token level!
>
> For instance, if we have a push with 5000 targeted devices, we are now
> able to say that 5 tokens, for instances failed, but APNs was happy to
> accept push request for the other 4995 devices (Note: this does NOT mean
> they actually arrive at the device, just that apple accepted them for
> further processing).
>
> Now, this, for APNs, gives us much more flexiblity handling our metrics!
>
> In our code, here, we do read *each* token request from APNs in here:
> https://github.com/aerogear/aerogear-unifiedpush-serve
> r/blob/20831d96196663349c96da6b5fe11aef65cacf59/push/sender/
> src/main/java/org/jboss/aerogear/unifiedpush/message/
> sender/apns/PushyApnsSender.java#L130-L147
>
> So here, we could simply send the result, on a per token base, to a
> (Kafka) topic, like:
>
> ...
> if (pushNotificationResponse.isAccepted()) {
> logger.trace("Push notification for '{}' (payload={})",
deviceToken, pushNotificationResponse.getPushNotification().getPayload());
>
> producer.send(jobID, "Success"); // sends to "push_messages"
topic
> } else {
> final String rejectReason = pushNotificationResponse.getRejectionReason();
> logger.trace("Push Message has been rejected with reason: {}",
rejectReason);
>
> producer.send(jobID, "Rejected"); // sends "push_messages"
topic
> ...
> }
>
> Now, this sends all to one topic, and we could be using, somewhere, Kafka
> Stream API, to perform some processing of the source, and calculate some
> stats on that, like:
>
> KStreamBuilder builder = new KStreamBuilder();
>
> // read from the topic that contains all messages, for all jobs
> final KStream<String, String> source =
builder.stream("push_messages");
>
>
> // some simple processing, and grouping by key, applying a predicate and send to
three "analytic" topic:
>
> final KTable<String, Long> successCountsPerJob = source.filter((key, value)
-> value.equals("Success"))
> .groupByKey()
> .count("successMessagesPerJob");
> successCountsPerJob.to(Serdes.String(), Serdes.Long(),
"successMessagesPerJob");
>
> final KTable<String, Long> failCountsPerJob = source.filter((key, value) ->
value.equals("Rejected"))
> .groupByKey()
> .count("failedMessagesPerJob");
> failCountsPerJob.to(Serdes.String(), Serdes.Long(),
"failedMessagesPerJob");
>
> source.groupByKey()
> count("totalMessagesPerJob")
> .to(Serdes.String(), Serdes.Long(), "totalMessagesPerJob");
>
>
> The above performs some functional processing of the single source of
> truth, based on different assumptions.
>
> If one would have a simple consumer on each of these three "analytic"
> topics, a simple logging output would be:
>
> 2017-05-16 13:42:48,763 INFO successMessagesPerJob: 2 - jobID: XXX
> 2017-05-16 13:42:48,764 INFO totalMessagesPerJob: 3 - jobID: XXX
> 2017-05-16 13:42:48,764 INFO failedMessagesPerJob: 1 - jobID: XXX
>
>
> since for the GSoC we do have two students, working on Kafka and HBase
> improvements for UPS, I wanted to share this quick prototype, as food for
> thoughts.
>
> Of course, each of these 'filtered' consumers could than eventually store
> the result somewhere else.
>
> With this approach, Kafka would be come the hub (or data pipeline) for
> our metrics, with stream processing and different consumers to deal with
> the results of interest
>
> Any comments or other thoughts?
>
> -Matthias
>
> --
> Matthias Wessendorf
>
> blog: http://matthiaswessendorf.wordpress.com/
> twitter: http://twitter.com/mwessendorf
>
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
--
JOSE MIGUEL GALLAS OLMEDO
ASSOCIATE QE, mobile
Red Hat
<https://www.redhat.com/>
M: +34618488633 <http://redhatemailsignature-marketing.itos.redhat.com/>
<https://red.ht/sig>
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
twitter: http://twitter.com/mwessendorf
6:50 p.m.
Such great news.
It would be fantastic if failing APN connection or similar would fire off a webhook. So
its possible for integrated systems can be notified.
BEST REGARDS
__________________
MADS MØLLER
CTO, PARTNER
Napp A/S
T: +45 42 42 80 60
M: +45 20 28 20 26
E: mm@napp.dk<mailto:mm@napp.dk>
W: https://napp.dk<https://napp.dk/>
__________________
From: <aerogear-dev-bounces(a)lists.jboss.org> on behalf of Matthias Wessendorf
<matzew(a)apache.org>
Reply-To: AeroGear Developer Mailing List <aerogear-dev(a)lists.jboss.org>
Date: Tuesday, 16 May 2017 at 16.50
To: AeroGear Developer Mailing List <aerogear-dev(a)lists.jboss.org>
Subject: Re: [aerogear-dev] Notification Delivery metrics and processing with Kafka
Oh, this is far from being done :-)
I just did a little POC, and since we also have two GSoC students, we have some time to
define the behavior, including UI, here on the commiunity :)
On Tue, May 16, 2017 at 3:37 PM, Jose Miguel Gallas Olmedo
<jgallaso@redhat.com<mailto:jgallaso@redhat.com>> wrote:
Great news!!
Will this information be displayed in the UI? As a tooltip or when extending the row in
"activity log"s table.
On 16 May 2017 at 13:58, Matthias Wessendorf
<matzew@apache.org<mailto:matzew@apache.org>> wrote:
Hi,
with the new APNs HTTP/2 APIs, and our usage of Pushy, we are able to get a way more
finegrain knowledge if Apple did accept (for further processing) or reject a messages, on
a per device_token level!
For instance, if we have a push with 5000 targeted devices, we are now able to say that 5
tokens, for instances failed, but APNs was happy to accept push request for the other 4995
devices (Note: this does NOT mean they actually arrive at the device, just that apple
accepted them for further processing).
Now, this, for APNs, gives us much more flexiblity handling our metrics!
In our code, here, we do read each token request from APNs in here:
https://github.com/aerogear/aerogear-unifiedpush-server/blob/20831d961966...
So here, we could simply send the result, on a per token base, to a (Kafka) topic, like:
...
if (pushNotificationResponse.isAccepted()) {
logger.trace("Push notification for '{}' (payload={})", deviceToken,
pushNotificationResponse.getPushNotification().getPayload());
producer.send(jobID, "Success"); // sends to "push_messages" topic
} else {
final String rejectReason = pushNotificationResponse.getRejectionReason();
logger.trace("Push Message has been rejected with reason: {}",
rejectReason);
producer.send(jobID, "Rejected"); // sends "push_messages" topic
...
}
Now, this sends all to one topic, and we could be using, somewhere, Kafka Stream API, to
perform some processing of the source, and calculate some stats on that, like:
KStreamBuilder builder = new KStreamBuilder();
// read from the topic that contains all messages, for all jobs
final KStream<String, String> source = builder.stream("push_messages");
// some simple processing, and grouping by key, applying a predicate and send to three
"analytic" topic:
final KTable<String, Long> successCountsPerJob = source.filter((key, value) ->
value.equals("Success"))
.groupByKey()
.count("successMessagesPerJob");
successCountsPerJob.to(Serdes.String(), Serdes.Long(),
"successMessagesPerJob");
final KTable<String, Long> failCountsPerJob = source.filter((key, value) ->
value.equals("Rejected"))
.groupByKey()
.count("failedMessagesPerJob");
failCountsPerJob.to(Serdes.String(), Serdes.Long(), "failedMessagesPerJob");
source.groupByKey()
count("totalMessagesPerJob")
.to(Serdes.String(), Serdes.Long(), "totalMessagesPerJob");
The above performs some functional processing of the single source of truth, based on
different assumptions.
If one would have a simple consumer on each of these three "analytic" topics, a
simple logging output would be:
2017-05-16 13:42:48,763 INFO successMessagesPerJob: 2 - jobID: XXX
2017-05-16 13:42:48,764 INFO totalMessagesPerJob: 3 - jobID: XXX
2017-05-16 13:42:48,764 INFO failedMessagesPerJob: 1 - jobID: XXX
since for the GSoC we do have two students, working on Kafka and HBase improvements for
UPS, I wanted to share this quick prototype, as food for thoughts.
Of course, each of these 'filtered' consumers could than eventually store the
result somewhere else.
With this approach, Kafka would be come the hub (or data pipeline) for our metrics, with
stream processing and different consumers to deal with the results of interest
Any comments or other thoughts?
-Matthias
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
twitter: http://twitter.com/mwessendorf
_______________________________________________
aerogear-dev mailing list
aerogear-dev@lists.jboss.org<mailto:aerogear-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/aerogear-dev
--
JOSE MIGUEL GALLAS OLMEDO
ASSOCIATE QE, mobile
Red Hat
<https://www.redhat.com/>
M: +34618488633<http://redhatemailsignature-marketing.itos.redhat.com/>
[https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-b...
_______________________________________________
aerogear-dev mailing list
aerogear-dev@lists.jboss.org<mailto:aerogear-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/aerogear-dev
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
twitter: http://twitter.com/mwessendorf
8:04 p.m.
right!
in theory, you can also simply register a custom kafka consumer to those
topics and apply some custom logic, e.g. to ping other systems
On Tue, May 16, 2017 at 6:50 PM, Mads Møller <mm(a)napp.dk> wrote:
Such great news.
It would be fantastic if failing APN connection or similar would fire off
a webhook. So its possible for integrated systems can be notified.
BEST REGARDS
*__________________*
*MADS MØLLER*
CTO, PARTNER
Napp A/S
T: +45 42 42 80 60 <+45%2042%2042%2080%2060>
M: +45 20 28 20 26 <+45%2020%2028%2020%2026>
E: mm(a)napp.dk
W: https://napp.dk
*__________________*
*From: *<aerogear-dev-bounces(a)lists.jboss.org> on behalf of Matthias
Wessendorf <matzew(a)apache.org>
*Reply-To: *AeroGear Developer Mailing List <aerogear-dev(a)lists.jboss.org>
*Date: *Tuesday, 16 May 2017 at 16.50
*To: *AeroGear Developer Mailing List <aerogear-dev(a)lists.jboss.org>
*Subject: *Re: [aerogear-dev] Notification Delivery metrics and
processing with Kafka
Oh, this is far from being done :-)
I just did a little POC, and since we also have two GSoC students, we have
some time to define the behavior, including UI, here on the commiunity :)
On Tue, May 16, 2017 at 3:37 PM, Jose Miguel Gallas Olmedo <
jgallaso(a)redhat.com> wrote:
Great news!!
Will this information be displayed in the UI? As a tooltip or when
extending the row in "activity log"s table.
On 16 May 2017 at 13:58, Matthias Wessendorf <matzew(a)apache.org> wrote:
Hi,
with the new APNs HTTP/2 APIs, and our usage of Pushy, we are able to get
a way more finegrain knowledge if Apple did accept (for further processing)
or reject a messages, on a per device_token level!
For instance, if we have a push with 5000 targeted devices, we are now
able to say that 5 tokens, for instances failed, but APNs was happy to
accept push request for the other 4995 devices (Note: this does NOT mean
they actually arrive at the device, just that apple accepted them for
further processing).
Now, this, for APNs, gives us much more flexiblity handling our metrics!
In our code, here, we do read *each* token request from APNs in here:
https://github.com/aerogear/aerogear-unifiedpush-server/blob/
20831d96196663349c96da6b5fe11aef65cacf59/push/sender/src/
main/java/org/jboss/aerogear/unifiedpush/message/sender/
apns/PushyApnsSender.java#L130-L147
So here, we could simply send the result, on a per token base, to a
(Kafka) topic, like:
...
if (pushNotificationResponse.isAccepted()) {
logger.trace("Push notification for '{}' (payload={})", deviceToken,
pushNotificationResponse.getPushNotification().getPayload());
producer.send(jobID, "Success"); // sends to "push_messages" topic
} else {
final String rejectReason = pushNotificationResponse.getRejectionReason();
logger.trace("Push Message has been rejected with reason: {}",
rejectReason);
producer.send(jobID, "Rejected"); // sends "push_messages" topic
...
}
Now, this sends all to one topic, and we could be using, somewhere, Kafka
Stream API, to perform some processing of the source, and calculate some
stats on that, like:
KStreamBuilder builder = new KStreamBuilder();
// read from the topic that contains all messages, for all jobs
final KStream<String, String> source = builder.stream("push_messages");
// some simple processing, and grouping by key, applying a predicate and send to three
"analytic" topic:
final KTable<String, Long> successCountsPerJob = source.filter((key, value) ->
value.equals("Success"))
.groupByKey()
.count("successMessagesPerJob");
successCountsPerJob.to(Serdes.String(), Serdes.Long(),
"successMessagesPerJob");
final KTable<String, Long> failCountsPerJob = source.filter((key, value) ->
value.equals("Rejected"))
.groupByKey()
.count("failedMessagesPerJob");
failCountsPerJob.to(Serdes.String(), Serdes.Long(), "failedMessagesPerJob");
source.groupByKey()
count("totalMessagesPerJob")
.to(Serdes.String(), Serdes.Long(), "totalMessagesPerJob");
The above performs some functional processing of the single source of
truth, based on different assumptions.
If one would have a simple consumer on each of these three "analytic"
topics, a simple logging output would be:
2017-05-16 13:42:48,763 INFO successMessagesPerJob: 2 - jobID: XXX
2017-05-16 13:42:48,764 INFO totalMessagesPerJob: 3 - jobID: XXX
2017-05-16 13:42:48,764 INFO failedMessagesPerJob: 1 - jobID: XXX
since for the GSoC we do have two students, working on Kafka and HBase
improvements for UPS, I wanted to share this quick prototype, as food for
thoughts.
Of course, each of these 'filtered' consumers could than eventually store
the result somewhere else.
With this approach, Kafka would be come the hub (or data pipeline) for our
metrics, with stream processing and different consumers to deal with the
results of interest
Any comments or other thoughts?
-Matthias
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
twitter: http://twitter.com/mwessendorf
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
--
*JOSE MIGUEL GALLAS OLMEDO*
ASSOCIATE QE, mobile
Red Hat
<https://www.redhat.com/>
M: +34618488633 <http://redhatemailsignature-marketing.itos.redhat.com/>
<https://red.ht/sig>
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
twitter: http://twitter.com/mwessendorf
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
twitter: http://twitter.com/mwessendorf
7:27 a.m.
New subject: Notification Delivery metrics and processing with Kafka
Are there Any guide/turorial on setting this up?
MED VENLIG HILSEN / BEST REGARDS
__________________
MADS MØLLER
CTO, PARTNER
Napp A/S
T: 42 42 80 60
M: 20 28 20 26
E: mm@napp.dk<mailto:mm@napp.dk>
W: www.napp.dk<http://www.napp.dk>
__________________
On 16 May 2017, at 21.46, Matthias Wessendorf
<matzew@apache.org<mailto:matzew@apache.org>> wrote:
right!
in theory, you can also simply register a custom kafka consumer to those topics and apply
some custom logic, e.g. to ping other systems
On Tue, May 16, 2017 at 6:50 PM, Mads Møller <mm@napp.dk<mailto:mm@napp.dk>>
wrote:
Such great news.
It would be fantastic if failing APN connection or similar would fire off a webhook. So
its possible for integrated systems can be notified.
BEST REGARDS
__________________
MADS MØLLER
CTO, PARTNER
Napp A/S
T: +45 42 42 80 60<tel:+45%2042%2042%2080%2060>
M: +45 20 28 20 26<tel:+45%2020%2028%2020%2026>
E: mm@napp.dk<mailto:mm@napp.dk>
W: https://napp.dk<https://napp.dk/>
__________________
From:
<aerogear-dev-bounces@lists.jboss.org<mailto:aerogear-dev-bounces@lists.jboss.org>>
on behalf of Matthias Wessendorf
<matzew@apache.org<mailto:matzew@apache.org>>
Reply-To: AeroGear Developer Mailing List
<aerogear-dev@lists.jboss.org<mailto:aerogear-dev@lists.jboss.org>>
Date: Tuesday, 16 May 2017 at 16.50
To: AeroGear Developer Mailing List
<aerogear-dev@lists.jboss.org<mailto:aerogear-dev@lists.jboss.org>>
Subject: Re: [aerogear-dev] Notification Delivery metrics and processing with Kafka
Oh, this is far from being done :-)
I just did a little POC, and since we also have two GSoC students, we have some time to
define the behavior, including UI, here on the commiunity :)
On Tue, May 16, 2017 at 3:37 PM, Jose Miguel Gallas Olmedo
<jgallaso@redhat.com<mailto:jgallaso@redhat.com>> wrote:
Great news!!
Will this information be displayed in the UI? As a tooltip or when extending the row in
"activity log"s table.
On 16 May 2017 at 13:58, Matthias Wessendorf
<matzew@apache.org<mailto:matzew@apache.org>> wrote:
Hi,
with the new APNs HTTP/2 APIs, and our usage of Pushy, we are able to get a way more
finegrain knowledge if Apple did accept (for further processing) or reject a messages, on
a per device_token level!
For instance, if we have a push with 5000 targeted devices, we are now able to say that 5
tokens, for instances failed, but APNs was happy to accept push request for the other 4995
devices (Note: this does NOT mean they actually arrive at the device, just that apple
accepted them for further processing).
Now, this, for APNs, gives us much more flexiblity handling our metrics!
In our code, here, we do read each token request from APNs in here:
https://github.com/aerogear/aerogear-unifiedpush-server/blob/20831d961966...
So here, we could simply send the result, on a per token base, to a (Kafka) topic, like:
...
if (pushNotificationResponse.isAccepted()) {
logger.trace("Push notification for '{}' (payload={})", deviceToken,
pushNotificationResponse.getPushNotification().getPayload());
producer.send(jobID, "Success"); // sends to "push_messages" topic
} else {
final String rejectReason = pushNotificationResponse.getRejectionReason();
logger.trace("Push Message has been rejected with reason: {}", rejectReason);
producer.send(jobID, "Rejected"); // sends "push_messages" topic
...
}
Now, this sends all to one topic, and we could be using, somewhere, Kafka Stream API, to
perform some processing of the source, and calculate some stats on that, like:
KStreamBuilder builder = new KStreamBuilder();
// read from the topic that contains all messages, for all jobs
final KStream<String, String> source = builder.stream("push_messages");
// some simple processing, and grouping by key, applying a predicate and send to three
"analytic" topic:
final KTable<String, Long> successCountsPerJob = source.filter((key, value) ->
value.equals("Success"))
.groupByKey()
.count("successMessagesPerJob");
successCountsPerJob.to(Serdes.String(), Serdes.Long(),
"successMessagesPerJob");
final KTable<String, Long> failCountsPerJob = source.filter((key, value) ->
value.equals("Rejected"))
.groupByKey()
.count("failedMessagesPerJob");
failCountsPerJob.to(Serdes.String(), Serdes.Long(), "failedMessagesPerJob");
source.groupByKey()
count("totalMessagesPerJob")
.to(Serdes.String(), Serdes.Long(), "totalMessagesPerJob");
The above performs some functional processing of the single source of truth, based on
different assumptions.
If one would have a simple consumer on each of these three "analytic" topics, a
simple logging output would be:
2017-05-16 13:42:48,763 INFO successMessagesPerJob: 2 - jobID: XXX
2017-05-16 13:42:48,764 INFO totalMessagesPerJob: 3 - jobID: XXX
2017-05-16 13:42:48,764 INFO failedMessagesPerJob: 1 - jobID: XXX
since for the GSoC we do have two students, working on Kafka and HBase improvements for
UPS, I wanted to share this quick prototype, as food for thoughts.
Of course, each of these 'filtered' consumers could than eventually store the
result somewhere else.
With this approach, Kafka would be come the hub (or data pipeline) for our metrics, with
stream processing and different consumers to deal with the results of interest
Any comments or other thoughts?
-Matthias
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
twitter: http://twitter.com/mwessendorf
_______________________________________________
aerogear-dev mailing list
aerogear-dev@lists.jboss.org<mailto:aerogear-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/aerogear-dev
--
JOSE MIGUEL GALLAS OLMEDO
ASSOCIATE QE, mobile
Red Hat
<https://www.redhat.com/>
M: +34618488633<http://redhatemailsignature-marketing.itos.redhat.com/>
[https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-b...
_______________________________________________
aerogear-dev mailing list
aerogear-dev@lists.jboss.org<mailto:aerogear-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/aerogear-dev
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
twitter: http://twitter.com/mwessendorf
_______________________________________________
aerogear-dev mailing list
aerogear-dev@lists.jboss.org<mailto:aerogear-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/aerogear-dev
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
twitter: http://twitter.com/mwessendorf
_______________________________________________
aerogear-dev mailing list
aerogear-dev@lists.jboss.org<mailto:aerogear-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/aerogear-dev
11:58 a.m.
we have nothing yet - this is just getting started.
Over the summer we have two students, as part of the Google Summer of Code,
working on this.
I just did a POC to evaluate the idea ;-)
On Wed, May 17, 2017 at 7:27 AM, Mads Møller <mm(a)napp.dk> wrote:
Are there Any guide/turorial on setting this up?
MED VENLIG HILSEN / BEST REGARDS
__________________
MADS MØLLER
CTO, PARTNER
Napp A/S
T: 42 42 80 60
M: 20 28 20 26
E: mm(a)napp.dk
W: www.napp.dk
__________________
On 16 May 2017, at 21.46, Matthias Wessendorf <matzew(a)apache.org> wrote:
right!
in theory, you can also simply register a custom kafka consumer to those
topics and apply some custom logic, e.g. to ping other systems
On Tue, May 16, 2017 at 6:50 PM, Mads Møller <mm(a)napp.dk> wrote:
> Such great news.
>
>
>
> It would be fantastic if failing APN connection or similar would fire off
> a webhook. So its possible for integrated systems can be notified.
>
>
>
>
>
>
>
> BEST REGARDS
>
> *__________________*
>
> *MADS MØLLER*
>
> CTO, PARTNER
>
>
>
> Napp A/S
>
> T: +45 42 42 80 60 <+45%2042%2042%2080%2060>
>
> M: +45 20 28 20 26 <+45%2020%2028%2020%2026>
>
> E: mm(a)napp.dk
>
> W: https://napp.dk
>
> *__________________*
>
>
>
>
>
>
>
>
>
>
>
> *From: *<aerogear-dev-bounces(a)lists.jboss.org> on behalf of Matthias
> Wessendorf <matzew(a)apache.org>
> *Reply-To: *AeroGear Developer Mailing List <aerogear-dev(a)lists.jboss.org
> >
> *Date: *Tuesday, 16 May 2017 at 16.50
> *To: *AeroGear Developer Mailing List <aerogear-dev(a)lists.jboss.org>
> *Subject: *Re: [aerogear-dev] Notification Delivery metrics and
> processing with Kafka
>
>
>
> Oh, this is far from being done :-)
>
>
>
> I just did a little POC, and since we also have two GSoC students, we
> have some time to define the behavior, including UI, here on the commiunity
> :)
>
>
>
> On Tue, May 16, 2017 at 3:37 PM, Jose Miguel Gallas Olmedo <
> jgallaso(a)redhat.com> wrote:
>
> Great news!!
>
>
>
> Will this information be displayed in the UI? As a tooltip or when
> extending the row in "activity log"s table.
>
>
>
> On 16 May 2017 at 13:58, Matthias Wessendorf <matzew(a)apache.org> wrote:
>
> Hi,
>
> with the new APNs HTTP/2 APIs, and our usage of Pushy, we are able to get
> a way more finegrain knowledge if Apple did accept (for further processing)
> or reject a messages, on a per device_token level!
>
> For instance, if we have a push with 5000 targeted devices, we are now
> able to say that 5 tokens, for instances failed, but APNs was happy to
> accept push request for the other 4995 devices (Note: this does NOT mean
> they actually arrive at the device, just that apple accepted them for
> further processing).
>
> Now, this, for APNs, gives us much more flexiblity handling our metrics!
>
> In our code, here, we do read *each* token request from APNs in here:
> https://github.com/aerogear/aerogear-unifiedpush-serve
> r/blob/20831d96196663349c96da6b5fe11aef65cacf59/push/sender/
> src/main/java/org/jboss/aerogear/unifiedpush/message/
> sender/apns/PushyApnsSender.java#L130-L147
>
> So here, we could simply send the result, on a per token base, to a
> (Kafka) topic, like:
>
> ...
>
> if (pushNotificationResponse.isAccepted()) {
>
> logger.trace("Push notification for '{}' (payload={})",
deviceToken, pushNotificationResponse.getPushNotification().getPayload());
>
>
>
> producer.send(jobID, "Success"); // sends to "push_messages"
topic
>
> } else {
>
> final String rejectReason = pushNotificationResponse.getRejectionReason();
>
> logger.trace("Push Message has been rejected with reason: {}",
rejectReason);
>
>
>
> producer.send(jobID, "Rejected"); // sends "push_messages"
topic
>
> ...
>
> }
>
> Now, this sends all to one topic, and we could be using, somewhere, Kafka
> Stream API, to perform some processing of the source, and calculate some
> stats on that, like:
>
> KStreamBuilder builder = new KStreamBuilder();
>
>
>
> // read from the topic that contains all messages, for all jobs
>
> final KStream<String, String> source =
builder.stream("push_messages");
>
>
>
>
>
> // some simple processing, and grouping by key, applying a predicate and send to
three "analytic" topic:
>
>
>
> final KTable<String, Long> successCountsPerJob = source.filter((key, value)
-> value.equals("Success"))
>
> .groupByKey()
>
> .count("successMessagesPerJob");
>
> successCountsPerJob.to(Serdes.String(), Serdes.Long(),
"successMessagesPerJob");
>
>
>
> final KTable<String, Long> failCountsPerJob = source.filter((key, value) ->
value.equals("Rejected"))
>
> .groupByKey()
>
> .count("failedMessagesPerJob");
>
> failCountsPerJob.to(Serdes.String(), Serdes.Long(),
"failedMessagesPerJob");
>
>
>
> source.groupByKey()
>
> count("totalMessagesPerJob")
>
> .to(Serdes.String(), Serdes.Long(), "totalMessagesPerJob");
>
>
>
> The above performs some functional processing of the single source of
> truth, based on different assumptions.
>
> If one would have a simple consumer on each of these three "analytic"
> topics, a simple logging output would be:
>
> 2017-05-16 13:42:48,763 INFO successMessagesPerJob: 2 - jobID: XXX
>
> 2017-05-16 13:42:48,764 INFO totalMessagesPerJob: 3 - jobID: XXX
>
> 2017-05-16 13:42:48,764 INFO failedMessagesPerJob: 1 - jobID: XXX
>
>
>
> since for the GSoC we do have two students, working on Kafka and HBase
> improvements for UPS, I wanted to share this quick prototype, as food for
> thoughts.
>
> Of course, each of these 'filtered' consumers could than eventually store
> the result somewhere else.
>
> With this approach, Kafka would be come the hub (or data pipeline) for
> our metrics, with stream processing and different consumers to deal with
> the results of interest
>
> Any comments or other thoughts?
>
> -Matthias
>
>
>
> --
>
> Matthias Wessendorf
>
> blog: http://matthiaswessendorf.wordpress.com/
> twitter: http://twitter.com/mwessendorf
>
>
>
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
>
>
>
>
> --
>
> *JOSE MIGUEL GALLAS OLMEDO*
>
> ASSOCIATE QE, mobile
>
> Red Hat
>
> <https://www.redhat.com/>
>
> M: +34618488633 <http://redhatemailsignature-marketing.itos.redhat.com/>
>
>
> <https://red.ht/sig>
>
>
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
>
>
>
>
> --
>
> Matthias Wessendorf
>
> blog: http://matthiaswessendorf.wordpress.com/
> twitter: http://twitter.com/mwessendorf
>
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
twitter: http://twitter.com/mwessendorf
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
twitter: http://twitter.com/mwessendorf
2536
days inactive
2537
days old
6 comments
3 participants
participants (3)
-
Jose Miguel Gallas Olmedo -
Mads Møller -
Matthias Wessendorf