[Hawkular-dev] Performance Tests of Different Hawkular Inventory Backends

[Lukas Krejci]

Titan does not use the lightweight transactions of Cassandra, but 
instead uses a locking mechanism of its own making [1]. We use some of 
those mechanisms to ensure the consistency but frankly I've not done too 
much research into all the failure scenarios that might happen (this is 
definitely something we'd like to add to our perf tests, hint hint 
jkremser ;) ).

IMHO the change in the performance I've seen in the perf tests after the 
fixes in Inventory have more to do with caching of data that Titan does 
within a transaction boundary than some perf improvements gained from 
the backends.

[1] http://s3.thinkaurelius.com/docs/titan/0.5.0/eventual-consistency.html

On 02/23/2016 08:53 PM, John Sanda wrote:
> Lukas do you have any insight into how Titan supports transactions with
> Cassandra? I know that Titan still uses the thrift API,which presumably
> rules out the light weight transactions introduced in C* 2.0. These are
> not to be confused with ACID transactions. It is more of an atomic
> update using a consensus protocol. Then you have atomic or logged
> batches which are atomic in the sense that either all of the statements
> will succeed or none will; however, they can succeed *eventually*. These
> have more overhead than unlogged batches because the mutations are first
> written to a batch log. Applying multiple mutations to the same
> partitions is an atomic operation. In other words, whether you update 1
> or 50 columns, if it is done within the same partition (and within the
> same operation), it is atomic.
>
> My point being that Cassandra is definitely not a transactional data
> store, so I am really curious about what Titan is doing.
>
>> On Feb 23, 2016, at 1:48 PM, Lukas Krejci <lkrejci at redhat.com
>> <mailto:lkrejci at redhat.com>> wrote:
>>
>>
>>
>> On 02/23/2016 07:42 PM, Jay Shaughnessy wrote:
>>>
>>> Lukas,
>>>
>>> That's excellent news.  Multiple backends is not something we really
>>> want to deal with.  Also, it might be nice to see a short presentation
>>> on the "best practices" for Tx handling.  But then again, that Tx stuff
>>> is handled at the Gremlin level?  So, perhaps not relevant to direct C*
>>> consumers like Alerts.
>>>
>>
>> You're right. Inventory uses Gremlin to handle transactions, so it
>> doesn't directly "see" what is Titan doing behind the scenes.
>>
>>> On 2/23/2016 12:43 PM, Lukas Krejci wrote:
>>>> Hi all,
>>>>
>>>> lately I've become really dissatisfied with how Inventory performed and
>>>> semi-publicly blamed Titan for that (because that was what looked like
>>>> the cause of all world's problems in my then uneducated eyes ;) ).
>>>>
>>>> I decided to do some performance comparisons. Because we didn't want
>>>> Hawkular to ship with 2 different NoSQL backends (C* for metrics and
>>>> whatever else for Inventory), I chose an RDBMS as a good conservative
>>>> alternative (because people, IMHO, are still more comfortable dealing
>>>> with an RDBMS than with NoSQL databases).
>>>>
>>>> Currently, inventory is written against the graph DSL called Gremlin
>>>> (from Tinkerpop 2.6.0). Fortunately, there exists a "toy" SQL backend
>>>> for Tinkerpop 2 that we could try and see if it performed any good
>>>> (which would frankly be surprising, given the fact it stores the graph
>>>> data rather naively). With some luck, no code would have to be changed
>>>> on our side to see the results.
>>>>
>>>> We had no such luck.
>>>>
>>>> Making the inventory run with the SQL backend was literally a day worth
>>>> of work (if that) and the first preliminary tests showed that Inventory
>>>> with Postgres backend performed much much better that Titan with
>>>> embedded Cassandra. But the tests also uncovered some problems with the
>>>> way Inventory code handled transactions.
>>>>
>>>> Fast forward 3 weeks and see large parts of Hawkular inventory updated
>>>> to correctly handle transactions. Now a single call to Inventory really
>>>> results in at most 1 transaction in the backend.
>>>>
>>>> So, I went and re-ran the tests. Also, I refrained from using embedded
>>>> Cassandra and instead use a locally running 2-node cluster.
>>>>
>>>> The results caught me by surprise. Not so much that the naive SQL
>>>> backend didn't perform particularly well, but the difference between the
>>>> performance of Titan before and after the transaction handling fixes.
>>>>
>>>> To not keep you waiting any longer for the results: Titan + C* is the
>>>> winner.
>>>>
>>>> For nice charts that include comparison to the old misbehaving impl,
>>>> see:
>>>> https://dashboards.ly/ua-tALzrY9rEoRBXvsLXbZJHT
>>>>
>>>> Cheers,
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> hawkular-dev mailing list
>>> hawkular-dev at lists.jboss.org <mailto:hawkular-dev at lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>>
>>
>> --
>> Lukas Krejci
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev at lists.jboss.org <mailto:hawkular-dev at lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>

-- 
Lukas Krejci