[Hawkular-dev] [Update] Storing Inventory data in Cassandra

[Lukas Krejci]

tl;dr - This is gonna be fun (in the worst sense of the word)

We had a very good discussion with Stefan and John from the metrics team and I 
think we identified the main problematic areas of the cassandra backend and 
brainstormed possible solutions to them.

1) Inconsistencies
Today, inventory guarantees that if some entity has a certain sync hash, the 
users can be sure that the entity and all its children are in a certain state 
(names, configurations, defined operations, etc, all exactly match). This is 
also used to identify identical stuff across feeds (i.e. if find all identical 
types defined by various feeds and operate on them as if they were a single 
type - /traversal/f;feed/rt;Type/identical/rl;defines/type=resource). This can 
no longer be guaranteed in Cassandra. We could try to overcome this by trying 
several aproaches:

a) using a "staging" area for updates (copy current state, apply changes and 
then "replace" the live area with staging) but that essentially means 
implementing serialized transactions on top of an eventually consistent 
storage - something I am not completely ecstatic about given the manpower and 
time constraints we have.

b) Essentially considering C* as a blob store and just dump serialized 
(portions of) the graph to it, with all the processing being done in memory on 
the inventory server. This still means we have to implement transactional 
behavior on our own (albeit in memory, not in C*) and it still means that 
stored data could conflict if inventory was clustered.

c) Give up consistent sync and just write everything all the time. 
Inconsistencies will arise because sync doesn't touch external relationships 
(feed doesn't know that some glue code discovered that a war is part of 
"something bigger", be it a cluster, a logical app, whatever). At that point 
we can also outright get rid of the hashes, because they will never be 
guaranteed to be consistent. This means that we will no longer be able to tell 
whether two feeds define the same resource types, because that depends on the 
rt's having the same hash.

2) Performant Traversals
Right now I am trying to implement a naive approach to graph traversals where 
each "hop" between nodes of the graph is represented by (at least one) query 
(possibly there can be very many queries for a single hop if it is required to 
retrieve results per every incoming vertex in the traversal). This has been 
identified as a potential performance problem.

The only "remedy" suggested for this was to consider 1b) - just store the 
whole portions of the graph as a "blob" in C* and do the processing in-memory.
This scares me a little bit because it opens up many possibilities for 
operating on stale/incorrect data, raises the question of how to "partition" 
the graph (more granularly than by tenant) and at the same time avoid 
complexity of handling inter-partition relationships, etc.

3) Conclusion
We will start with a naive implementation with no guarantees of consistency 
and will try to identify the concrete problematic areas of the code (the above 
already hints at some we assume will cause problems). Then we will try to 
modify the implementation/storage model/functionality/guarantees iteratively 
to fix the concrete problems identified.

Lukas

On Wednesday, November 30, 2016 12:08:05 PM CET theute at redhat.com wrote:

I can't join today, Heiko neither.
Feel free to go ahead with the call though and please send a feedback on 
hawkular-dev.

Storing Inventory data in Cassandra
Currently, we're storing inventory data in an SQL database. Metrics on the 
other hand store the data in Cassandra. We're exploring how to unify the 
storage backends for Hawkular components and hence the title.
We'll use https://docs.google.com/document/d/1Lgv8WE1j0r7rir5hTpV-xutKFChyoNPEH3XSz020ayk as a starting point for the discussion.
To join the Meeting:
https://bluejeans.com/8169978803
To join via Browser:
https://bluejeans.com/8169978803/browser
To join with Lync:
https://bluejeans.com/8169978803/lync
To join via Room System:
Video Conferencing System: bjn.vc -or-199.48.152.152
Meeting ID : 8169978803
To join via phone :
1)  Dial: 
	+44 203 574 6870
	(see all numbers - https://www.intercallonline.com/
listNumbersByCode.action?confCode=8169978803)
2)  Enter Conference ID : 8169978803

When
Wed Nov 30, 2016 3pm – 4pm Zurich
Where
https://bluejeans.com/8169978803 (map)
Who

•
lkrejci at redhat.com - creator
•
jsanda at redhat.com
•
jtakvori at redhat.com
•
hawkular-dev at lists.jboss.org






-- 
Lukas Krejci