<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I’d like to expose something in the REST API.<div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Sep 6, 2016, at 3:43 AM, Thomas Heute <<a href="mailto:theute@redhat.com" class="">theute@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Agreed.<div class=""><br class=""></div><div class="">What user interface do you have in mind here ? CLI ? JMX ? WebUI ?</div><div class=""><br class=""></div><div class=""><br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Fri, Sep 2, 2016 at 5:34 PM, John Sanda <span dir="ltr" class=""><<a href="mailto:jsanda@redhat.com" target="_blank" class="">jsanda@redhat.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">To date we haven’t really done anything by way of managing/monitoring the Cassandra cluster. We need to monitor Cassandra in order to know things like:<br class="">
<br class="">
* When additional nodes are needed<br class="">
* When disk space is low<br class="">
* When I/O is too slow<br class="">
* When more heap space is needed<br class="">
<br class="">
Cassandra exposes a lot of metrics. I created HWKMETRICS-448. It briefly talks about collecting metrics from Cassandra. In terms of managing the cluster, I will provide a few concrete examples that have come up recently in OpenShift.<br class="">
<br class="">
Scenario 1: User deploys additional node(s) to reduce the load on cluster<br class="">
After the new node has bootstrapped and is running, we need to run nodetool cleanup on each node (or run it via JMX) in order to remove keys/data that each each node no longer owns; otherwise, disk space won’t be freed up. The cleanup operation can potentially be resource intensive as it triggers compactions. Given this, we probably want to run it one node at a time. Right now the user is left to do this manually.<br class="">
<br class="">
Scenario 2: User deploys additional node(s) to get replication and fault tolerance<br class="">
I connect to Cassandra directly via cqlsh and update replication_factor. I then need to run repair on each node can be tricky because 1) it is resource intensive, 2) can take a long time, 3) prone to failure, and 4) Cassandra does not give progress indicators.<br class="">
<br class="">
Scenario 3: User sets up regularly, scheduled repair to ensure data is consistent across cluster<br class="">
Once replication_factor > 1, repair needs to be run on a regular basis. More specifically it should be run within gc_grace_seconds which is configured per table and defaults to 10 days. The data table in metrics has reduced gc_grace_seconds to 1 day and probably reduce it to zero since it is append-only. The value for gc_grace_seconds might vary per table based on access patterns, which means the frequency of repair should vary as well.<br class="">
<br class="">
<br class="">
There has already been some discussion of these things for Hawkular Metrics in the context of OpenShift. It applies to all of Hawkular Services as well. Initially I was thinking about building some management components directly in metrics, but it probably makes more sense as a separate, shared component (or components) that can be reused in both stand alone metrics in OpenShift and a full Hawkular Services deployment in MiQ for example.<br class="">
<br class="">
We are already running into these scenarios in OpenShift and probably need to start putting something in place sooner rather than later.<br class="">
______________________________<wbr class="">_________________<br class="">
hawkular-dev mailing list<br class="">
<a href="mailto:hawkular-dev@lists.jboss.org" class="">hawkular-dev@lists.jboss.org</a><br class="">
<a href="https://lists.jboss.org/mailman/listinfo/hawkular-dev" rel="noreferrer" target="_blank" class="">https://lists.jboss.org/<wbr class="">mailman/listinfo/hawkular-dev</a><br class="">
<br class="">
<br class="">
</blockquote></div><br class=""></div>
_______________________________________________<br class="">hawkular-dev mailing list<br class=""><a href="mailto:hawkular-dev@lists.jboss.org" class="">hawkular-dev@lists.jboss.org</a><br class="">https://lists.jboss.org/mailman/listinfo/hawkular-dev<br class=""></div></blockquote></div><br class=""></div></body></html>