<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Our process was pretty straight
forward. We (ok, I) picked memcache for three reasons: <br>
1) Automatic expiration of records means I don't have to do my own
garbage collection. <br>
2) AWS offered ElastiCache which was built off of Memcache and
could allow us to move to Couchbase if needed (it's not)<br>
3) Our Ops folk suggested Memcache because they've had the least
number of problems dealing with it. <br>
<br>
Since I started working on things, a few other things have
happened:<br>
1) AWS now offers a managed Redis service<br>
2) AWS became a bit more reliable in a subsequent release.
(Haven't done a lot of rigorous checks there, since most of the
problems come about because Redis REALLY doesn't like it when
memory gets full.)<br>
3) Some of the storage requirements changed. <br>
<br>
Ah, the joys of initial implementations of a service. This is why
abstraction is a dear, sweet friend of mine. <br>
<br>
Honestly, the thing I'm having the hardest time trying to work out
is the optimal way that records should be stored and accessed in
the various data stores. As anyone with a link to
<a class="moz-txt-link-freetext" href="http://www.eecs.berkeley.edu/~rcs/research/interactive_latency.html">http://www.eecs.berkeley.edu/~rcs/research/interactive_latency.html</a>
can tell you, pulling stuff off a wire is one of the worst things
you can do. So, the question is "Do you pull lots of data in one
block, or lots of blocks of little data?" Either way, there's
going to be a factor of inefficiency. (e.g. someone who has a lot
of CHIDs with only a one or two very active CHIDs would have a
pretty good sized block of extra data pumped back and forth in the
case of a single record organized by UAID, vs. someone with a lot
of very active CHIDs having to do lots of little updates.)<br>
<br>
I'm not really going to try to address it right now, since it's
premature optimization without actual use metrics, and there be
both dragons and hairy yaks there, but I tend to like having
options on hand for when things start to go wonky.<br>
<br>
I'll be interested to see what y'all pick, since your constraints
and benefits aren't going to be the same. <br>
<br>
On 2013/9/25 5:16 AM, Daniel Bevenius wrote:<br>
</div>
<blockquote
cite="mid:CAFJm-3HpMe+JXJf71RNxuCeCkiXjpRys4U+0obGbRS0H1Hbo_w@mail.gmail.com"
type="cite">
<div dir="ltr">To get a feel for what would be involved to use a
key/value datastore we've done some experimenting with Redis[1].
There might be other non-relational databases more suited or
perhaps Redis is a good choice for this, I don't know. But I
think we should decide if this is worth pursuing and in that
case what database to use before spending more time on this.
<div>
Let us know what you think.</div>
<div><br>
</div>
<div>[1] <a moz-do-not-send="true"
href="https://gist.github.com/danbev/6606289#using-redis-as-a-data-store">https://gist.github.com/danbev/6606289#using-redis-as-a-data-store</a></div>
</div>
<div class="gmail_extra">
<br>
<br>
<div class="gmail_quote">On 19 September 2013 18:08, JR Conlin <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:jrconlin@gmail.com" target="_blank">jrconlin@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div class="im">
<div>On 2013/9/19 5:18 AM, Lucas Holmquist wrote:<br>
</div>
<blockquote type="cite"> <br>
<div>
<div>On Sep 19, 2013, at 12:34 AM, Daniel Bevenius
<<a moz-do-not-send="true"
href="mailto:daniel.bevenius@gmail.com"
target="_blank">daniel.bevenius@gmail.com</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">><span
style="font-family:arial,sans-serif;font-size:13px">I
wonder what kind of numbers would we get by
ditching JPA completely </span><span
style="font-family:arial,sans-serif;font-size:13px">and
using a non-relational DB like Redis</span>
<div> <span
style="font-family:arial,sans-serif;font-size:13px">Yeah,
I think we will most likely need to if we
want to come close to the other
implementations performance wise. Others use
Memcache and I've seen MongoDB in use as
well. </span></div>
<div><span
style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<div><span
style="font-family:arial,sans-serif;font-size:13px">Perhaps
I should just add performance tests for the
rest of the SimplePush operations so that we
have them covered and then look into using a
non-relational DB. Once that is done we can
revisit this performance task.</span></div>
<div><span
style="font-family:arial,sans-serif;font-size:13px">What
do people thing about that?</span></div>
</div>
</blockquote>
<div><br>
</div>
<div>+1, relational DB's are dinosaours <br>
</div>
</div>
</blockquote>
</div>
Hardly. It's just a question of what the right tool for a
given job is. (I'll note that Google is spending quite a
bit of time and effort improving Maria because they use a
LOT of relational DBs for very large data.<br>
<br>
In this case, however, it's pretty easy to reduce things
to simple key/value. I picked Memcache, partly because of
the fact that it does record auto-expiration, which means
that I don't have to do garbage collection on uncollected
records. If you switched to an alternate schema (such as
keeping a single record per UAID that contained all the
CHID data as well as stuff like the proprietary info or
other crap), you could even use simple flat files and skip
the DB requirement altogether. <br>
<br>
We were kicking the idea around of only storing
undeliverable data into the DB, and instead letting each
websock connector deal with managing it's own data. For
our implementation, I've already preferenced delivery over
storage for connected clients and seen a fair bit of
improvement on delivery. (Remember, SimplePush is not a
100% guaranteed delivery system, so please avoid using it
for nuclear reactor management or pacemakers.)<br>
<br>
We'll probably hold off on doing further memory refinement
until we get some actual use data, but I like having
options available. <br>
<div class="im">
<blockquote type="cite">
<div><br>
<blockquote type="cite">
<div dir="ltr">
<div><span
style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<div><span
style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On 19 September 2013
06:03, Bruno Oliveira <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:bruno@abstractj.org"
target="_blank">bruno@abstractj.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">Hmmm tempting
idea :)<br>
<div>
<div><br>
> On Sep 19, 2013, at 12:23 AM,
Douglas Campos <<a
moz-do-not-send="true"
href="mailto:qmx@qmx.me"
target="_blank">qmx@qmx.me</a>>
wrote:<br>
><br>
> That's a nice report!<br>
><br>
> I wonder what kind of numbers would
we get by ditching JPA completely<br>
> and using a non-relational DB like
Redis...<br>
><br>
> --<br>
> qmx<br>
</div>
</div>
>
_______________________________________________<br>
> aerogear-dev mailing list<br>
> <a moz-do-not-send="true"
href="mailto:aerogear-dev@lists.jboss.org"
target="_blank">aerogear-dev@lists.jboss.org</a><br>
> <a moz-do-not-send="true"
href="https://lists.jboss.org/mailman/listinfo/aerogear-dev"
target="_blank">https://lists.jboss.org/mailman/listinfo/aerogear-dev</a><br>
_______________________________________________<br>
aerogear-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:aerogear-dev@lists.jboss.org"
target="_blank">aerogear-dev@lists.jboss.org</a><br>
<a moz-do-not-send="true"
href="https://lists.jboss.org/mailman/listinfo/aerogear-dev"
target="_blank">https://lists.jboss.org/mailman/listinfo/aerogear-dev</a><br>
</blockquote>
</div>
<br>
</div>
_______________________________________________<br>
aerogear-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:aerogear-dev@lists.jboss.org"
target="_blank">aerogear-dev@lists.jboss.org</a><br>
<a moz-do-not-send="true"
href="https://lists.jboss.org/mailman/listinfo/aerogear-dev"
target="_blank">https://lists.jboss.org/mailman/listinfo/aerogear-dev</a></blockquote>
</div>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
aerogear-dev mailing list
<a moz-do-not-send="true" href="mailto:aerogear-dev@lists.jboss.org" target="_blank">aerogear-dev@lists.jboss.org</a>
<a moz-do-not-send="true" href="https://lists.jboss.org/mailman/listinfo/aerogear-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/aerogear-dev</a></pre>
</blockquote>
<br>
</div>
</div>
<br>
_______________________________________________<br>
aerogear-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:aerogear-dev@lists.jboss.org">aerogear-dev@lists.jboss.org</a><br>
<a moz-do-not-send="true"
href="https://lists.jboss.org/mailman/listinfo/aerogear-dev"
target="_blank">https://lists.jboss.org/mailman/listinfo/aerogear-dev</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
aerogear-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:aerogear-dev@lists.jboss.org">aerogear-dev@lists.jboss.org</a>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/aerogear-dev">https://lists.jboss.org/mailman/listinfo/aerogear-dev</a></pre>
</blockquote>
<br>
</body>
</html>