5:07 a.m.
I've begun looking into AGPUSH-291 to try to identify areas to start
improving the performance of our SimplePush Server. The following gist can
be used to follow this work:
https://gist.github.com/danbev/6606289
Any comments can be posted here or as comments on the gist.
11:23 p.m.
That's a nice report!
I wonder what kind of numbers would we get by ditching JPA completely
and using a non-relational DB like Redis...
--
qmx
12:03 a.m.
Hmmm tempting idea :)
On Sep 19, 2013, at 12:23 AM, Douglas Campos <qmx(a)qmx.me>
wrote:
That's a nice report!
I wonder what kind of numbers would we get by ditching JPA completely
and using a non-relational DB like Redis...
--
qmx
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
12:34 a.m.
I wonder what kind of numbers would we get by ditching JPA completely
and
using a non-relational DB like Redis
Yeah, I think we will most likely need to if we want to come close to the
other implementations performance wise. Others use Memcache and I've seen
MongoDB in use as well.
Perhaps I should just add performance tests for the rest of the SimplePush
operations so that we have them covered and then look into using a
non-relational DB. Once that is done we can revisit this performance task.
What do people thing about that?
On 19 September 2013 06:03, Bruno Oliveira <bruno(a)abstractj.org> wrote:
Hmmm tempting idea :)
> On Sep 19, 2013, at 12:23 AM, Douglas Campos <qmx(a)qmx.me> wrote:
>
> That's a nice report!
>
> I wonder what kind of numbers would we get by ditching JPA completely
> and using a non-relational DB like Redis...
>
> --
> qmx
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
12:37 a.m.
On Sep 18, 2013, at 11:34 PM, Daniel Bevenius <daniel.bevenius(a)gmail.com> wrote:
>I wonder what kind of numbers would we get by ditching JPA
completely and using a non-relational DB like Redis
Yeah, I think we will most likely need to if we want to come close to the other
implementations performance wise. Others use Memcache and I've seen MongoDB in use as
well.
Perhaps I should just add performance tests for the rest of the SimplePush operations so
that we have them covered and then look into using a non-relational DB. Once that is done
we can revisit this performance task.
What do people thing about that?
+1 to all of that
On 19 September 2013 06:03, Bruno Oliveira <bruno(a)abstractj.org> wrote:
Hmmm tempting idea :)
> On Sep 19, 2013, at 12:23 AM, Douglas Campos <qmx(a)qmx.me> wrote:
>
> That's a nice report!
>
> I wonder what kind of numbers would we get by ditching JPA completely
> and using a non-relational DB like Redis...
>
> --
> qmx
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
1:09 a.m.
On Thu, Sep 19, 2013 at 6:34 AM, Daniel Bevenius
<daniel.bevenius(a)gmail.com>wrote:
>I wonder what kind of numbers would we get by ditching JPA
completely and
using a non-relational DB like Redis
Yeah, I think we will most likely need to if we want to come close to the
other implementations performance wise. Others use Memcache and I've seen
MongoDB in use as well.
Perhaps I should just add performance tests for the rest of the SimplePush
operations so that we have them covered and then look into using a
non-relational DB. Once that is done we can revisit this performance task.
What do people thing about that?
That sounds like a good plan!
-M
On 19 September 2013 06:03, Bruno Oliveira <bruno(a)abstractj.org> wrote:
> Hmmm tempting idea :)
>
> > On Sep 19, 2013, at 12:23 AM, Douglas Campos <qmx(a)qmx.me> wrote:
> >
> > That's a nice report!
> >
> > I wonder what kind of numbers would we get by ditching JPA completely
> > and using a non-relational DB like Redis...
> >
> > --
> > qmx
> > _______________________________________________
> > aerogear-dev mailing list
> > aerogear-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/aerogear-dev
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
--
Matthias Wessendorf
blog: http://matthiaswessendorf.wordpress.com/
sessions: http://www.slideshare.net/mwessendorf
twitter: http://twitter.com/mwessendorf
2:10 a.m.
On Thu, Sep 19, 2013 at 6:34 AM, Daniel Bevenius <daniel.bevenius(a)gmail.com>
wrote:
>I wonder what kind of numbers would we get by ditching JPA completely and using a
non-relational DB like Redis
Yeah, I think we will most likely need to if we want to come close to the other
implementations performance wise. Others use Memcache and I've seen MongoDB in use as
well.
Perhaps I should just add performance tests for the rest of the SimplePush operations so
that we have them covered and then look into using a non-relational DB. Once that is done
we can revisit this performance task.
What do people thing about that?
Why not do something simpler first and validate that by adding a cache on the database
access?
2:31 a.m.
Why not do something simpler first and validate that by adding a
cache on
the database access?
That is also an option but I'm not sure how well that would scale in the
long run. I think that we will later need to be able to host multiple
instances in an environment like OpenShift where I'm not sure if a
relational database is the best choice. Having a relational database might
not really be needed either, but was chosen mainly because that was the
most familiar and quickest solution at the time to have persistence of some
sort.
On 19 September 2013 08:10, Erik Jan de Wit <edewit(a)redhat.com> wrote:
>
> On Thu, Sep 19, 2013 at 6:34 AM, Daniel Bevenius <
> daniel.bevenius(a)gmail.com> wrote:
>
>> >I wonder what kind of numbers would we get by ditching JPA completely and
>> using a non-relational DB like Redis
>> Yeah, I think we will most likely need to if we want to come close to
>> the other implementations performance wise. Others use Memcache and I've
>> seen MongoDB in use as well.
>>
>> Perhaps I should just add performance tests for the rest of the
>> SimplePush operations so that we have them covered and then look into using
>> a non-relational DB. Once that is done we can revisit this performance task.
>> What do people thing about that?
>>
>
>
Why not do something simpler first and validate that by adding a
cache on
> the database access?
>
>
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
7:24 a.m.
On Sep 19, 2013, at 1:31 AM, Daniel Bevenius <daniel.bevenius(a)gmail.com> wrote:
> Why not do something simpler first and validate that by adding a
cache on the database access?
That is also an option but I'm not sure how well that would scale in the long run. I
think that we will later need to be able to host multiple instances in an environment like
OpenShift where I'm not sure if a relational database is the best choice. Having a
relational database might not really be needed either, but was chosen mainly because that
was the most familiar and quickest solution at the time to have persistence of some sort.
I agree. We should not spend time on trying to speed up a relational database when we are
very likely not going to use one in the future and it doesn't seem like the right fit
for this implementation anyway.
On 19 September 2013 08:10, Erik Jan de Wit <edewit(a)redhat.com> wrote:
>
> On Thu, Sep 19, 2013 at 6:34 AM, Daniel Bevenius <daniel.bevenius(a)gmail.com>
wrote:
> >I wonder what kind of numbers would we get by ditching JPA completely and using a
non-relational DB like Redis
> Yeah, I think we will most likely need to if we want to come close to the other
implementations performance wise. Others use Memcache and I've seen MongoDB in use as
well.
>
> Perhaps I should just add performance tests for the rest of the SimplePush operations
so that we have them covered and then look into using a non-relational DB. Once that is
done we can revisit this performance task.
> What do people thing about that?
>
Why not do something simpler first and validate that by adding a cache on the database
access?
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
8:18 a.m.
On Sep 19, 2013, at 12:34 AM, Daniel Bevenius <daniel.bevenius(a)gmail.com> wrote:
>I wonder what kind of numbers would we get by ditching JPA
completely and using a non-relational DB like Redis
Yeah, I think we will most likely need to if we want to come close to the other
implementations performance wise. Others use Memcache and I've seen MongoDB in use as
well.
Perhaps I should just add performance tests for the rest of the SimplePush operations so
that we have them covered and then look into using a non-relational DB. Once that is done
we can revisit this performance task.
What do people thing about that?
+1, relational DB's are dinosaours
On 19 September 2013 06:03, Bruno Oliveira <bruno(a)abstractj.org> wrote:
Hmmm tempting idea :)
> On Sep 19, 2013, at 12:23 AM, Douglas Campos <qmx(a)qmx.me> wrote:
>
> That's a nice report!
>
> I wonder what kind of numbers would we get by ditching JPA completely
> and using a non-relational DB like Redis...
>
> --
> qmx
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
12:08 p.m.
On 2013/9/19 5:18 AM, Lucas Holmquist wrote:
On Sep 19, 2013, at 12:34 AM, Daniel Bevenius
<daniel.bevenius(a)gmail.com <mailto:daniel.bevenius@gmail.com>> wrote:
> >I wonder what kind of numbers would we get by ditching JPA
> completely and using a non-relational DB like Redis
> Yeah, I think we will most likely need to if we want to come close to
> the other implementations performance wise. Others use Memcache and
> I've seen MongoDB in use as well.
>
> Perhaps I should just add performance tests for the rest of the
> SimplePush operations so that we have them covered and then look into
> using a non-relational DB. Once that is done we can revisit this
> performance task.
> What do people thing about that?
+1, relational DB's are dinosaours
Hardly. It's just a question of what
the right tool for a given job is.
(I'll note that Google is spending quite a bit of time and effort
improving Maria because they use a LOT of relational DBs for very large
data.
In this case, however, it's pretty easy to reduce things to simple
key/value. I picked Memcache, partly because of the fact that it does
record auto-expiration, which means that I don't have to do garbage
collection on uncollected records. If you switched to an alternate
schema (such as keeping a single record per UAID that contained all the
CHID data as well as stuff like the proprietary info or other crap), you
could even use simple flat files and skip the DB requirement altogether.
We were kicking the idea around of only storing undeliverable data into
the DB, and instead letting each websock connector deal with managing
it's own data. For our implementation, I've already preferenced delivery
over storage for connected clients and seen a fair bit of improvement on
delivery. (Remember, SimplePush is not a 100% guaranteed delivery
system, so please avoid using it for nuclear reactor management or
pacemakers.)
We'll probably hold off on doing further memory refinement until we get
some actual use data, but I like having options available.
>
>
>
>
> On 19 September 2013 06:03, Bruno Oliveira <bruno(a)abstractj.org
> <mailto:bruno@abstractj.org>> wrote:
>
> Hmmm tempting idea :)
>
> > On Sep 19, 2013, at 12:23 AM, Douglas Campos <qmx(a)qmx.me
> <mailto:qmx@qmx.me>> wrote:
> >
> > That's a nice report!
> >
> > I wonder what kind of numbers would we get by ditching JPA
> completely
> > and using a non-relational DB like Redis...
> >
> > --
> > qmx
> > _______________________________________________
> > aerogear-dev mailing list
> > aerogear-dev(a)lists.jboss.org <mailto:aerogear-dev@lists.jboss.org>
> > https://lists.jboss.org/mailman/listinfo/aerogear-dev
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org <mailto:aerogear-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
>
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org <mailto:aerogear-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
8:16 a.m.
To get a feel for what would be involved to use a key/value datastore we've
done some experimenting with Redis[1]. There might be other non-relational
databases more suited or perhaps Redis is a good choice for this, I don't
know. But I think we should decide if this is worth pursuing and in that
case what database to use before spending more time on this.
Let us know what you think.
[1] https://gist.github.com/danbev/6606289#using-redis-as-a-data-store
On 19 September 2013 18:08, JR Conlin <jrconlin(a)gmail.com> wrote:
On 2013/9/19 5:18 AM, Lucas Holmquist wrote:
On Sep 19, 2013, at 12:34 AM, Daniel Bevenius <daniel.bevenius(a)gmail.com>
wrote:
>I wonder what kind of numbers would we get by ditching JPA completely and
using a non-relational DB like Redis
Yeah, I think we will most likely need to if we want to come close to the
other implementations performance wise. Others use Memcache and I've seen
MongoDB in use as well.
Perhaps I should just add performance tests for the rest of the
SimplePush operations so that we have them covered and then look into using
a non-relational DB. Once that is done we can revisit this performance task.
What do people thing about that?
+1, relational DB's are dinosaours
Hardly. It's just a question of what the right tool for a given job is.
(I'll note that Google is spending quite a bit of time and effort improving
Maria because they use a LOT of relational DBs for very large data.
In this case, however, it's pretty easy to reduce things to simple
key/value. I picked Memcache, partly because of the fact that it does
record auto-expiration, which means that I don't have to do garbage
collection on uncollected records. If you switched to an alternate schema
(such as keeping a single record per UAID that contained all the CHID data
as well as stuff like the proprietary info or other crap), you could even
use simple flat files and skip the DB requirement altogether.
We were kicking the idea around of only storing undeliverable data into
the DB, and instead letting each websock connector deal with managing it's
own data. For our implementation, I've already preferenced delivery over
storage for connected clients and seen a fair bit of improvement on
delivery. (Remember, SimplePush is not a 100% guaranteed delivery system,
so please avoid using it for nuclear reactor management or pacemakers.)
We'll probably hold off on doing further memory refinement until we get
some actual use data, but I like having options available.
On 19 September 2013 06:03, Bruno Oliveira <bruno(a)abstractj.org> wrote:
> Hmmm tempting idea :)
>
> > On Sep 19, 2013, at 12:23 AM, Douglas Campos <qmx(a)qmx.me> wrote:
> >
> > That's a nice report!
> >
> > I wonder what kind of numbers would we get by ditching JPA completely
> > and using a non-relational DB like Redis...
> >
> > --
> > qmx
> > _______________________________________________
> > aerogear-dev mailing list
> > aerogear-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/aerogear-dev
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
_______________________________________________
aerogear-dev mailing
listaerogear-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/aerogear-dev
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
11:35 a.m.
Our process was pretty straight forward. We (ok, I) picked memcache for
three reasons:
1) Automatic expiration of records means I don't have to do my own
garbage collection.
2) AWS offered ElastiCache which was built off of Memcache and could
allow us to move to Couchbase if needed (it's not)
3) Our Ops folk suggested Memcache because they've had the least number
of problems dealing with it.
Since I started working on things, a few other things have happened:
1) AWS now offers a managed Redis service
2) AWS became a bit more reliable in a subsequent release. (Haven't done
a lot of rigorous checks there, since most of the problems come about
because Redis REALLY doesn't like it when memory gets full.)
3) Some of the storage requirements changed.
Ah, the joys of initial implementations of a service. This is why
abstraction is a dear, sweet friend of mine.
Honestly, the thing I'm having the hardest time trying to work out is
the optimal way that records should be stored and accessed in the
various data stores. As anyone with a link to
http://www.eecs.berkeley.edu/~rcs/research/interactive_latency.html can
tell you, pulling stuff off a wire is one of the worst things you can
do. So, the question is "Do you pull lots of data in one block, or lots
of blocks of little data?" Either way, there's going to be a factor of
inefficiency. (e.g. someone who has a lot of CHIDs with only a one or
two very active CHIDs would have a pretty good sized block of extra data
pumped back and forth in the case of a single record organized by UAID,
vs. someone with a lot of very active CHIDs having to do lots of little
updates.)
I'm not really going to try to address it right now, since it's
premature optimization without actual use metrics, and there be both
dragons and hairy yaks there, but I tend to like having options on hand
for when things start to go wonky.
I'll be interested to see what y'all pick, since your constraints and
benefits aren't going to be the same.
On 2013/9/25 5:16 AM, Daniel Bevenius wrote:
To get a feel for what would be involved to use a key/value
datastore
we've done some experimenting with Redis[1]. There might be other
non-relational databases more suited or perhaps Redis is a good choice
for this, I don't know. But I think we should decide if this is worth
pursuing and in that case what database to use before spending more
time on this.
Let us know what you think.
[1] https://gist.github.com/danbev/6606289#using-redis-as-a-data-store
On 19 September 2013 18:08, JR Conlin <jrconlin(a)gmail.com
<mailto:jrconlin@gmail.com>> wrote:
On 2013/9/19 5:18 AM, Lucas Holmquist wrote:
>
> On Sep 19, 2013, at 12:34 AM, Daniel Bevenius
> <daniel.bevenius(a)gmail.com <mailto:daniel.bevenius@gmail.com>>
wrote:
>
>> >I wonder what kind of numbers would we get by ditching JPA
>> completely and using a non-relational DB like Redis
>> Yeah, I think we will most likely need to if we want to come
>> close to the other implementations performance wise. Others use
>> Memcache and I've seen MongoDB in use as well.
>>
>> Perhaps I should just add performance tests for the rest of the
>> SimplePush operations so that we have them covered and then look
>> into using a non-relational DB. Once that is done we can revisit
>> this performance task.
>> What do people thing about that?
>
> +1, relational DB's are dinosaours
Hardly. It's just a question of what the right tool for a given
job is. (I'll note that Google is spending quite a bit of time and
effort improving Maria because they use a LOT of relational DBs
for very large data.
In this case, however, it's pretty easy to reduce things to simple
key/value. I picked Memcache, partly because of the fact that it
does record auto-expiration, which means that I don't have to do
garbage collection on uncollected records. If you switched to an
alternate schema (such as keeping a single record per UAID that
contained all the CHID data as well as stuff like the proprietary
info or other crap), you could even use simple flat files and skip
the DB requirement altogether.
We were kicking the idea around of only storing undeliverable data
into the DB, and instead letting each websock connector deal with
managing it's own data. For our implementation, I've already
preferenced delivery over storage for connected clients and seen a
fair bit of improvement on delivery. (Remember, SimplePush is not
a 100% guaranteed delivery system, so please avoid using it for
nuclear reactor management or pacemakers.)
We'll probably hold off on doing further memory refinement until
we get some actual use data, but I like having options available.
>
>>
>>
>>
>>
>> On 19 September 2013 06:03, Bruno Oliveira <bruno(a)abstractj.org
>> <mailto:bruno@abstractj.org>> wrote:
>>
>> Hmmm tempting idea :)
>>
>> > On Sep 19, 2013, at 12:23 AM, Douglas Campos <qmx(a)qmx.me
>> <mailto:qmx@qmx.me>> wrote:
>> >
>> > That's a nice report!
>> >
>> > I wonder what kind of numbers would we get by ditching JPA
>> completely
>> > and using a non-relational DB like Redis...
>> >
>> > --
>> > qmx
>> > _______________________________________________
>> > aerogear-dev mailing list
>> > aerogear-dev(a)lists.jboss.org
>> <mailto:aerogear-dev@lists.jboss.org>
>> > https://lists.jboss.org/mailman/listinfo/aerogear-dev
>> _______________________________________________
>> aerogear-dev mailing list
>> aerogear-dev(a)lists.jboss.org
>> <mailto:aerogear-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>
>>
>> _______________________________________________
>> aerogear-dev mailing list
>> aerogear-dev(a)lists.jboss.org <mailto:aerogear-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
>
>
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org <mailto:aerogear-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org <mailto:aerogear-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/aerogear-dev
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
12:17 p.m.
Looks promising, however Redis has limitation - all data are basically stored in
memory, requiring additional setup for enterprise application.
I was wondering whether SimplePush could use for instance Hibernate OGM as
abstraction, giving user possibility to switch DB storage, while Redis seems to
be reasonable default [1]? On OpenShift, it would for instance make sense to use
MongoDB, as this is already a cartridge available.
Karel
[1] This is an interesting comparison of NoSQL databases:
http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis
On Wed, 25 Sep 2013 14:16:30 +0200
Daniel Bevenius <daniel.bevenius(a)gmail.com> wrote:
To get a feel for what would be involved to use a key/value datastore
we've
done some experimenting with Redis[1]. There might be other non-relational
databases more suited or perhaps Redis is a good choice for this, I don't
know. But I think we should decide if this is worth pursuing and in that
case what database to use before spending more time on this.
Let us know what you think.
[1] https://gist.github.com/danbev/6606289#using-redis-as-a-data-store
On 19 September 2013 18:08, JR Conlin <jrconlin(a)gmail.com> wrote:
> On 2013/9/19 5:18 AM, Lucas Holmquist wrote:
>
>
> On Sep 19, 2013, at 12:34 AM, Daniel Bevenius <daniel.bevenius(a)gmail.com>
> wrote:
>
> >I wonder what kind of numbers would we get by ditching JPA completely and
> using a non-relational DB like Redis
> Yeah, I think we will most likely need to if we want to come close to the
> other implementations performance wise. Others use Memcache and I've seen
> MongoDB in use as well.
>
> Perhaps I should just add performance tests for the rest of the
> SimplePush operations so that we have them covered and then look into using
> a non-relational DB. Once that is done we can revisit this performance task.
> What do people thing about that?
>
>
> +1, relational DB's are dinosaours
>
> Hardly. It's just a question of what the right tool for a given job is.
> (I'll note that Google is spending quite a bit of time and effort improving
> Maria because they use a LOT of relational DBs for very large data.
>
> In this case, however, it's pretty easy to reduce things to simple
> key/value. I picked Memcache, partly because of the fact that it does
> record auto-expiration, which means that I don't have to do garbage
> collection on uncollected records. If you switched to an alternate schema
> (such as keeping a single record per UAID that contained all the CHID data
> as well as stuff like the proprietary info or other crap), you could even
> use simple flat files and skip the DB requirement altogether.
>
> We were kicking the idea around of only storing undeliverable data into
> the DB, and instead letting each websock connector deal with managing it's
> own data. For our implementation, I've already preferenced delivery over
> storage for connected clients and seen a fair bit of improvement on
> delivery. (Remember, SimplePush is not a 100% guaranteed delivery system,
> so please avoid using it for nuclear reactor management or pacemakers.)
>
> We'll probably hold off on doing further memory refinement until we get
> some actual use data, but I like having options available.
>
>
>
>
>
>
> On 19 September 2013 06:03, Bruno Oliveira <bruno(a)abstractj.org> wrote:
>
>> Hmmm tempting idea :)
>>
>> > On Sep 19, 2013, at 12:23 AM, Douglas Campos <qmx(a)qmx.me> wrote:
>> >
>> > That's a nice report!
>> >
>> > I wonder what kind of numbers would we get by ditching JPA completely
>> > and using a non-relational DB like Redis...
>> >
>> > --
>> > qmx
>> > _______________________________________________
>> > aerogear-dev mailing list
>> > aerogear-dev(a)lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/aerogear-dev
>> _______________________________________________
>> aerogear-dev mailing list
>> aerogear-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>
>
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
>
>
>
> _______________________________________________
> aerogear-dev mailing
>
listaerogear-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/aerogear-dev
>
>
>
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
12:53 p.m.
The concern with using Mongo is that it's very lossy and fairly
unreliable as a data store for a number of reasons. (It's fine for
simple, low demand systems, but has complications once you really start
to hammer on it.)
http://blog.schmichael.com/2011/11/05/failing-with-mongodb/
http://blog.engineering.kiip.me/post/20988881092/a-year-with-mongodb
etc.
I'd suggest sticking with other DBs unless you're ok with loss or don't
plan on heavily exercising it.
On 2013/9/25 9:17 AM, Karel Piwko wrote:
Looks promising, however Redis has limitation - all data are
basically stored in
memory, requiring additional setup for enterprise application.
I was wondering whether SimplePush could use for instance Hibernate OGM as
abstraction, giving user possibility to switch DB storage, while Redis seems to
be reasonable default [1]? On OpenShift, it would for instance make sense to use
MongoDB, as this is already a cartridge available.
Karel
[1] This is an interesting comparison of NoSQL databases:
http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis
On Wed, 25 Sep 2013 14:16:30 +0200
Daniel Bevenius <daniel.bevenius(a)gmail.com> wrote:
> To get a feel for what would be involved to use a key/value datastore we've
> done some experimenting with Redis[1]. There might be other non-relational
> databases more suited or perhaps Redis is a good choice for this, I don't
> know. But I think we should decide if this is worth pursuing and in that
> case what database to use before spending more time on this.
> Let us know what you think.
>
> [1] https://gist.github.com/danbev/6606289#using-redis-as-a-data-store
>
>
> On 19 September 2013 18:08, JR Conlin <jrconlin(a)gmail.com> wrote:
>
>> On 2013/9/19 5:18 AM, Lucas Holmquist wrote:
>>
>>
>> On Sep 19, 2013, at 12:34 AM, Daniel Bevenius <daniel.bevenius(a)gmail.com>
>> wrote:
>>
>> >I wonder what kind of numbers would we get by ditching JPA completely and
>> using a non-relational DB like Redis
>> Yeah, I think we will most likely need to if we want to come close to the
>> other implementations performance wise. Others use Memcache and I've seen
>> MongoDB in use as well.
>>
>> Perhaps I should just add performance tests for the rest of the
>> SimplePush operations so that we have them covered and then look into using
>> a non-relational DB. Once that is done we can revisit this performance task.
>> What do people thing about that?
>>
>>
>> +1, relational DB's are dinosaours
>>
>> Hardly. It's just a question of what the right tool for a given job is.
>> (I'll note that Google is spending quite a bit of time and effort improving
>> Maria because they use a LOT of relational DBs for very large data.
>>
>> In this case, however, it's pretty easy to reduce things to simple
>> key/value. I picked Memcache, partly because of the fact that it does
>> record auto-expiration, which means that I don't have to do garbage
>> collection on uncollected records. If you switched to an alternate schema
>> (such as keeping a single record per UAID that contained all the CHID data
>> as well as stuff like the proprietary info or other crap), you could even
>> use simple flat files and skip the DB requirement altogether.
>>
>> We were kicking the idea around of only storing undeliverable data into
>> the DB, and instead letting each websock connector deal with managing it's
>> own data. For our implementation, I've already preferenced delivery over
>> storage for connected clients and seen a fair bit of improvement on
>> delivery. (Remember, SimplePush is not a 100% guaranteed delivery system,
>> so please avoid using it for nuclear reactor management or pacemakers.)
>>
>> We'll probably hold off on doing further memory refinement until we get
>> some actual use data, but I like having options available.
>>
>>
>>
>>
>>
>>
>> On 19 September 2013 06:03, Bruno Oliveira <bruno(a)abstractj.org> wrote:
>>
>>> Hmmm tempting idea :)
>>>
>>>> On Sep 19, 2013, at 12:23 AM, Douglas Campos <qmx(a)qmx.me> wrote:
>>>>
>>>> That's a nice report!
>>>>
>>>> I wonder what kind of numbers would we get by ditching JPA completely
>>>> and using a non-relational DB like Redis...
>>>>
>>>> --
>>>> qmx
>>> > _______________________________________________
>>>> aerogear-dev mailing list
>>>> aerogear-dev(a)lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>> _______________________________________________
>>> aerogear-dev mailing list
>>> aerogear-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>>
>> _______________________________________________
>> aerogear-dev mailing list
>> aerogear-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>
>>
>>
>>
>> _______________________________________________
>> aerogear-dev mailing
>>
listaerogear-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/aerogear-dev
>>
>>
>>
>> _______________________________________________
>> aerogear-dev mailing list
>> aerogear-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
4:59 p.m.
On Wed, Sep 25, 2013 at 09:53:19AM -0700, JR Conlin wrote:
The concern with using Mongo is that it's very lossy and fairly
unreliable as a data store for a number of reasons. (It's fine for
simple, low demand systems, but has complications once you really start
to hammer on it.)
http://blog.schmichael.com/2011/11/05/failing-with-mongodb/
http://blog.engineering.kiip.me/post/20988881092/a-year-with-mongodb
etc.
I'd suggest sticking with other DBs unless you're ok with loss or don't
plan on heavily exercising it.
Seconded, have had severe data loss with mongo under high load at my
previous job
--
qmx
5:50 a.m.
No reason to push for Mongo then, my experience with NoSQL databases (not
counting XML databases) is pretty limited and I'm glad you gave me some insight.
What about abstraction layer idea based on Hibernate OGM? Does it make any
sense to you? Currently is does support only Infinispan, EHCache and MongoDB
[1], so I guess it's out of question as well.
Karel
[1]
http://docs.jboss.org/hibernate/ogm/4.0/reference/en-US/html/ogm-datastor...
On Wed, 25 Sep 2013 17:59:10 -0300
Douglas Campos <qmx(a)qmx.me> wrote:
On Wed, Sep 25, 2013 at 09:53:19AM -0700, JR Conlin wrote:
> The concern with using Mongo is that it's very lossy and fairly
> unreliable as a data store for a number of reasons. (It's fine for
> simple, low demand systems, but has complications once you really start
> to hammer on it.)
>
> http://blog.schmichael.com/2011/11/05/failing-with-mongodb/
> http://blog.engineering.kiip.me/post/20988881092/a-year-with-mongodb
> etc.
>
> I'd suggest sticking with other DBs unless you're ok with loss or don't
> plan on heavily exercising it.
Seconded, have had severe data loss with mongo under high load at my
previous job
6:27 a.m.
What about abstraction layer idea based on Hibernate OGM?
I
agree with being able to support different type of databases, relational
as well as non-relational. The interface is pretty simple but everything is
currently in the core modules which is bad. We have now extracted them into
separate modules[1] and I'm currently working on improving the interface
and adding the Redis datastore example into a separate module as well.
I've not looked into Hibernate OGM but if it is limited to those then
perhaps this is something for a future version.
[1] https://issues.jboss.org/browse/AGPUSH-363
On 26 September 2013 11:50, Karel Piwko <kpiwko(a)redhat.com> wrote:
No reason to push for Mongo then, my experience with NoSQL databases
(not
counting XML databases) is pretty limited and I'm glad you gave me some
insight.
What about abstraction layer idea based on Hibernate OGM? Does it make any
sense to you? Currently is does support only Infinispan, EHCache and
MongoDB
[1], so I guess it's out of question as well.
Karel
[1]
http://docs.jboss.org/hibernate/ogm/4.0/reference/en-US/html/ogm-datastor...
On Wed, 25 Sep 2013 17:59:10 -0300
Douglas Campos <qmx(a)qmx.me> wrote:
> On Wed, Sep 25, 2013 at 09:53:19AM -0700, JR Conlin wrote:
> > The concern with using Mongo is that it's very lossy and fairly
> > unreliable as a data store for a number of reasons. (It's fine for
> > simple, low demand systems, but has complications once you really start
> > to hammer on it.)
> >
> > http://blog.schmichael.com/2011/11/05/failing-with-mongodb/
> > http://blog.engineering.kiip.me/post/20988881092/a-year-with-mongodb
> > etc.
> >
> > I'd suggest sticking with other DBs unless you're ok with loss or
don't
> > plan on heavily exercising it.
>
> Seconded, have had severe data loss with mongo under high load at my
> previous job
>
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
11:15 a.m.
I'm not familiar with Hibernate, but I'm a HUGE advocate of abstraction
at functional levels. (e.g. classic MVC designs). Generally, what's good
today may not be good tomorrow, and having unencumbered interfaces lets
you swap in what makes the most amount of sense with the least impact to
your code.
The nice thing about Key/value is that you've got the most amount of
flexibility around what you actually use to store data; Postgres, MySQL,
Redis, Memcache, heck an in memory hash can all do it.
As for which data store you actually use, that's kinda driven by what
you expect your load to be. If you're only talking a few thousand
clients with a dozen channels, each getting an update when a flights
status changes, then you're not talking about putting a huge strain on
your storage or it's Read/Write capacity. If you've got a few hundred
million clients and some partners who've decided to use your system as
part of their FPS game updates, then you've got lots more problems.
(And now you start to understand why I wake up in a cold sweat screaming
about how you don't use SimplePush for real time stock quotes.)
On 2013/9/26 2:50 AM, Karel Piwko wrote:
No reason to push for Mongo then, my experience with NoSQL databases
(not
counting XML databases) is pretty limited and I'm glad you gave me some insight.
What about abstraction layer idea based on Hibernate OGM? Does it make any
sense to you? Currently is does support only Infinispan, EHCache and MongoDB
[1], so I guess it's out of question as well.
Karel
[1]
http://docs.jboss.org/hibernate/ogm/4.0/reference/en-US/html/ogm-datastor...
On Wed, 25 Sep 2013 17:59:10 -0300
Douglas Campos <qmx(a)qmx.me> wrote:
> On Wed, Sep 25, 2013 at 09:53:19AM -0700, JR Conlin wrote:
>> The concern with using Mongo is that it's very lossy and fairly
>> unreliable as a data store for a number of reasons. (It's fine for
>> simple, low demand systems, but has complications once you really start
>> to hammer on it.)
>>
>> http://blog.schmichael.com/2011/11/05/failing-with-mongodb/
>> http://blog.engineering.kiip.me/post/20988881092/a-year-with-mongodb
>> etc.
>>
>> I'd suggest sticking with other DBs unless you're ok with loss or
don't
>> plan on heavily exercising it.
> Seconded, have had severe data loss with mongo under high load at my
> previous job
>
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
4:50 a.m.
Sorry for my late reply to this thread, I've was side tracked with a few
other issues but I'm back on track now.
I don't have much to report at the moment except to share the branch we are
working on for the different database options, and how to run/configure
them if anyone would like to try them out.
We are currently looking into what store will be the best fit for our
SimplePush Server and are trying three types of databases, MySql (JPA),
Redis, and CouchDB in this
branch<https://github.com/danbev/aerogear-simplepush-server/tree/no-sq...
.
Using the above branch one can try the different database by making changes
to the AS7/WildFly configuration (standalone.xml in most cases).
For example, to use a JPA database one would use:
<subsystem xmlns="urn:org.jboss.aerogear.simplepush:1.0">
<server socket-binding="simplepush" token-key="changeMe!"
notification-tls="false" sockjs-tls="false">
<datastore>
<jpa
datasource-jndi-name="java:jboss/datasources/SimplePushDS"
persistence-unit="SimplePushPU"/>
</datastore>
</server></subsystem>
To use Redis:
<subsystem xmlns="urn:org.jboss.aerogear.simplepush:1.0">
<server socket-binding="simplepush" token-key="changeMe!"
notification-tls="false" sockjs-tls="false">
<datastore>
<redis host="localhost" port="6379"/>
</datastore>
</server></subsystem>
To use CouchDB:
<subsystem xmlns="urn:org.jboss.aerogear.simplepush:1.0">
<server socket-binding="simplepush" token-key="changeMe!"
notification-tls="false" sockjs-tls="false">
<datastore>
<couchdb url="http://127.0.0.1:5984"
database-name="simplepush"/>
</datastore>
</server></subsystem>
I'm writing a gist describing how data is stored with Redis and CouchDB and
will post back here when it's done to get feedback.
On 26 September 2013 17:15, JR Conlin <jrconlin(a)gmail.com> wrote:
I'm not familiar with Hibernate, but I'm a HUGE advocate of
abstraction
at functional levels. (e.g. classic MVC designs). Generally, what's good
today may not be good tomorrow, and having unencumbered interfaces lets
you swap in what makes the most amount of sense with the least impact to
your code.
The nice thing about Key/value is that you've got the most amount of
flexibility around what you actually use to store data; Postgres, MySQL,
Redis, Memcache, heck an in memory hash can all do it.
As for which data store you actually use, that's kinda driven by what
you expect your load to be. If you're only talking a few thousand
clients with a dozen channels, each getting an update when a flights
status changes, then you're not talking about putting a huge strain on
your storage or it's Read/Write capacity. If you've got a few hundred
million clients and some partners who've decided to use your system as
part of their FPS game updates, then you've got lots more problems.
(And now you start to understand why I wake up in a cold sweat screaming
about how you don't use SimplePush for real time stock quotes.)
On 2013/9/26 2:50 AM, Karel Piwko wrote:
> No reason to push for Mongo then, my experience with NoSQL databases (not
> counting XML databases) is pretty limited and I'm glad you gave me some
insight.
>
> What about abstraction layer idea based on Hibernate OGM? Does it make
any
> sense to you? Currently is does support only Infinispan, EHCache and
MongoDB
> [1], so I guess it's out of question as well.
>
> Karel
>
>
> [1]
>
http://docs.jboss.org/hibernate/ogm/4.0/reference/en-US/html/ogm-datastor...
>
> On Wed, 25 Sep 2013 17:59:10 -0300
> Douglas Campos <qmx(a)qmx.me> wrote:
>
>> On Wed, Sep 25, 2013 at 09:53:19AM -0700, JR Conlin wrote:
>>> The concern with using Mongo is that it's very lossy and fairly
>>> unreliable as a data store for a number of reasons. (It's fine for
>>> simple, low demand systems, but has complications once you really start
>>> to hammer on it.)
>>>
>>> http://blog.schmichael.com/2011/11/05/failing-with-mongodb/
>>> http://blog.engineering.kiip.me/post/20988881092/a-year-with-mongodb
>>> etc.
>>>
>>> I'd suggest sticking with other DBs unless you're ok with loss or
don't
>>> plan on heavily exercising it.
>> Seconded, have had severe data loss with mongo under high load at my
>> previous job
>>
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
_______________________________________________
aerogear-dev mailing list
aerogear-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-dev
8:13 a.m.
I've put together the following gist with a little information about how data
is being stored with Redis and CouchDB:
https://gist.github.com/danbev/6974640
--
View this message in context:
http://aerogear-dev.1069024.n5.nabble.com/aerogear-dev-simplepush-Perform...
Sent from the aerogear-dev mailing list archive at Nabble.com.
3845
days inactive
3871
days old
20 comments
10 participants
participants (10)
-
Bruno Oliveira -
Daniel Bevenius -
danielbevenius
-
Douglas Campos -
Erik Jan de Wit -
JR Conlin -
Karel Piwko -
Kris Borchers -
Lucas Holmquist -
Matthias Wessendorf