<div dir="ltr">To get a feel for what would be involved to use a key/value datastore we've done some experimenting with Redis[1]. There might be other non-relational databases more suited or perhaps Redis is a good choice for this, I don't know. But I think we should decide if this is worth pursuing and in that case what database to use before spending more time on this.<div>
Let us know what you think.</div><div><br></div><div>[1] <a href="https://gist.github.com/danbev/6606289#using-redis-as-a-data-store">https://gist.github.com/danbev/6606289#using-redis-as-a-data-store</a></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On 19 September 2013 18:08, JR Conlin <span dir="ltr"><<a href="mailto:jrconlin@gmail.com" target="_blank">jrconlin@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><div class="im">
<div>On 2013/9/19 5:18 AM, Lucas Holmquist
wrote:<br>
</div>
<blockquote type="cite">
<br>
<div>
<div>On Sep 19, 2013, at 12:34 AM, Daniel Bevenius <<a href="mailto:daniel.bevenius@gmail.com" target="_blank">daniel.bevenius@gmail.com</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">><span style="font-family:arial,sans-serif;font-size:13px">I
wonder what kind of numbers would we get by ditching JPA
completely </span><span style="font-family:arial,sans-serif;font-size:13px">and
using a non-relational DB like Redis</span>
<div>
<span style="font-family:arial,sans-serif;font-size:13px">Yeah,
I think we will most likely need to if we want to come
close to the other implementations performance wise.
Others use Memcache and I've seen MongoDB in use as
well. </span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px">Perhaps
I should just add performance tests for the rest of the
SimplePush operations so that we have them covered and
then look into using a non-relational DB. Once that is
done we can revisit this performance task.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px">What
do people thing about that?</span></div>
</div>
</blockquote>
<div><br>
</div>
<div>+1, relational DB's are dinosaours <br>
</div>
</div>
</blockquote></div>
Hardly. It's just a question of what the right tool for a given job
is. (I'll note that Google is spending quite a bit of time and
effort improving Maria because they use a LOT of relational DBs for
very large data.<br>
<br>
In this case, however, it's pretty easy to reduce things to simple
key/value. I picked Memcache, partly because of the fact that it
does record auto-expiration, which means that I don't have to do
garbage collection on uncollected records. If you switched to an
alternate schema (such as keeping a single record per UAID that
contained all the CHID data as well as stuff like the proprietary
info or other crap), you could even use simple flat files and skip
the DB requirement altogether. <br>
<br>
We were kicking the idea around of only storing undeliverable data
into the DB, and instead letting each websock connector deal with
managing it's own data. For our implementation, I've already
preferenced delivery over storage for connected clients and seen a
fair bit of improvement on delivery. (Remember, SimplePush is not a
100% guaranteed delivery system, so please avoid using it for
nuclear reactor management or pacemakers.)<br>
<br>
We'll probably hold off on doing further memory refinement until we
get some actual use data, but I like having options available. <br><div class="im">
<blockquote type="cite">
<div><br>
<blockquote type="cite">
<div dir="ltr">
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On 19 September 2013 06:03, Bruno
Oliveira <span dir="ltr"><<a href="mailto:bruno@abstractj.org" target="_blank">bruno@abstractj.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hmmm
tempting idea :)<br>
<div>
<div><br>
> On Sep 19, 2013, at 12:23 AM, Douglas Campos
<<a href="mailto:qmx@qmx.me" target="_blank">qmx@qmx.me</a>> wrote:<br>
><br>
> That's a nice report!<br>
><br>
> I wonder what kind of numbers would we get by
ditching JPA completely<br>
> and using a non-relational DB like Redis...<br>
><br>
> --<br>
> qmx<br>
</div>
</div>
> _______________________________________________<br>
> aerogear-dev mailing list<br>
> <a href="mailto:aerogear-dev@lists.jboss.org" target="_blank">aerogear-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/aerogear-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/aerogear-dev</a><br>
_______________________________________________<br>
aerogear-dev mailing list<br>
<a href="mailto:aerogear-dev@lists.jboss.org" target="_blank">aerogear-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/aerogear-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/aerogear-dev</a><br>
</blockquote>
</div>
<br>
</div>
_______________________________________________<br>
aerogear-dev mailing list<br>
<a href="mailto:aerogear-dev@lists.jboss.org" target="_blank">aerogear-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/aerogear-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/aerogear-dev</a></blockquote>
</div>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
aerogear-dev mailing list
<a href="mailto:aerogear-dev@lists.jboss.org" target="_blank">aerogear-dev@lists.jboss.org</a>
<a href="https://lists.jboss.org/mailman/listinfo/aerogear-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/aerogear-dev</a></pre>
</blockquote>
<br>
</div></div>
<br>_______________________________________________<br>
aerogear-dev mailing list<br>
<a href="mailto:aerogear-dev@lists.jboss.org">aerogear-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/aerogear-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/aerogear-dev</a><br></blockquote></div><br></div>