<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Sep 6, 2016, at 11:45 AM, Matt Wringe <<a href="mailto:mwringe@redhat.com" class="">mwringe@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">----- Original Message -----</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">From: "Michael Burman" <<a href="mailto:miburman@redhat.com" class="">miburman@redhat.com</a>><br class="">To: "Discussions around Hawkular development" <<a href="mailto:hawkular-dev@lists.jboss.org" class="">hawkular-dev@lists.jboss.org</a>><br class="">Sent: Tuesday, 6 September, 2016 11:09:45 AM<br class="">Subject: Re: [Hawkular-dev] managing cassandra cluster<br class=""><br class="">Hi,<br class=""><br class="">Well, actually I would say for Openshift we should try to hit a single<br class="">container strategy, which has both HWKMETRICS & Cassandra deployed. This is<br class="">to ensure that some of the operations we're going to do in the future can<br class="">take advantage of data locality.<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Hawkular Metrics and Cassandra are not cheap components to run in a cluster, they take up a lot of resources. If we can run multiple Cassandras for every 1 Hawkular (or vice versa) then we can use a lot less resources than if we always need to scale both up because one has become a bottleneck.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">I would really not want to bundle these together if we don't have to.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">What operations would require bundling them together exactly?</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">So unless we run a separate service inside the Cassandra containers, there's<br class="">no easy single metric to get from Cassandra that provides "needs scaling".<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">This is what I was expecting, I was not expecting Cassandra itself to provide a nice default metric to scale or not based on a specific value. We would have some service running along with Cassandra monitoring itself and the cluster to determine if scaling is required or not.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote><div><br class=""></div><div>Here’s what I had in mind at least for an initial effort. We package our own metrics reporter[1] with Cassandra. It pushes data to hawkular metrics. We provide a management endpoint that will indicate whether or not scaling is necessary based on the metrics we collect.</div><div><br class=""></div><div>[1] <a href="http://www.datastax.com/dev/blog/pluggable-metrics-reporting-in-cassandra-2-0-2" class="">http://www.datastax.com/dev/blog/pluggable-metrics-reporting-in-cassandra-2-0-2</a></div><br class=""><blockquote type="cite" class=""><div class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""> - Micke<br class=""><br class="">----- Original Message -----<br class="">From: "Matt Wringe" <<a href="mailto:mwringe@redhat.com" class="">mwringe@redhat.com</a>><br class="">To: "Discussions around Hawkular development" <<a href="mailto:hawkular-dev@lists.jboss.org" class="">hawkular-dev@lists.jboss.org</a>><br class="">Sent: Tuesday, September 6, 2016 5:26:07 PM<br class="">Subject: Re: [Hawkular-dev] managing cassandra cluster<br class=""><br class=""><br class=""><br class="">----- Original Message -----<br class=""><blockquote type="cite" class="">From: "John Sanda" <<a href="mailto:jsanda@redhat.com" class="">jsanda@redhat.com</a>><br class="">To: "Discussions around Hawkular development"<br class=""><<a href="mailto:hawkular-dev@lists.jboss.org" class="">hawkular-dev@lists.jboss.org</a>><br class="">Sent: Friday, 2 September, 2016 11:34:07 AM<br class="">Subject: [Hawkular-dev] managing cassandra cluster<br class=""><br class="">To date we haven’t really done anything by way of managing/monitoring the<br class="">Cassandra cluster. We need to monitor Cassandra in order to know things<br class="">like:<br class=""><br class="">* When additional nodes are needed<br class="">* When disk space is low<br class="">* When I/O is too slow<br class="">* When more heap space is needed<br class=""><br class="">Cassandra exposes a lot of metrics. I created HWKMETRICS-448. It briefly<br class="">talks about collecting metrics from Cassandra. In terms of managing the<br class="">cluster, I will provide a few concrete examples that have come up recently<br class="">in OpenShift.<br class=""><br class="">Scenario 1: User deploys additional node(s) to reduce the load on cluster<br class="">After the new node has bootstrapped and is running, we need to run nodetool<br class="">cleanup on each node (or run it via JMX) in order to remove keys/data that<br class="">each each node no longer owns; otherwise, disk space won’t be freed up. The<br class="">cleanup operation can potentially be resource intensive as it triggers<br class="">compactions. Given this, we probably want to run it one node at a time.<br class="">Right now the user is left to do this manually.<br class=""><br class="">Scenario 2: User deploys additional node(s) to get replication and fault<br class="">tolerance<br class="">I connect to Cassandra directly via cqlsh and update replication_factor. I<br class="">then need to run repair on each node can be tricky because 1) it is<br class="">resource<br class="">intensive, 2) can take a long time, 3) prone to failure, and 4) Cassandra<br class="">does not give progress indicators.<br class=""><br class="">Scenario 3: User sets up regularly, scheduled repair to ensure data is<br class="">consistent across cluster<br class="">Once replication_factor > 1, repair needs to be run on a regular basis.<br class="">More<br class="">specifically it should be run within gc_grace_seconds which is configured<br class="">per table and defaults to 10 days. The data table in metrics has reduced<br class="">gc_grace_seconds to 1 day and probably reduce it to zero since it is<br class="">append-only. The value for gc_grace_seconds might vary per table based on<br class="">access patterns, which means the frequency of repair should vary as well.<br class=""><br class=""><br class="">There has already been some discussion of these things for Hawkular Metrics<br class="">in the context of OpenShift. It applies to all of Hawkular Services as<br class="">well.<br class="">Initially I was thinking about building some management components directly<br class="">in metrics, but it probably makes more sense as a separate, shared<br class="">component<br class="">(or components) that can be reused in both stand alone metrics in OpenShift<br class="">and a full Hawkular Services deployment in MiQ for example.<br class=""></blockquote><br class="">On OpenShift, the ideal situation here would be to have the Cassandra<br class="">instances themselves expose a metric that we can use to determine when the<br class="">Cassandra cluster is under too much load and needs to scale up. The HPA<br class="">would then read this metric and automatically scale the cluster up if<br class="">needed.<br class=""><br class="">If we determine that cannot be done for whatever reason and that Hawkular<br class="">Metrics needs to determine when to scale or not, there are ways we can do<br class="">this. But it gets a little bit more tricky. If given the right permissions<br class="">we can go out to the cluster and do things like scale up components, perform<br class="">operations of the Cassandra containers directly, etc. Ideally the HPA should<br class="">be handling this, but we could get around it if absolutely needed.<br class=""><br class=""><blockquote type="cite" class=""><br class="">We are already running into these scenarios in OpenShift and probably need<br class="">to<br class="">start putting something in place sooner rather than later.<br class="">_______________________________________________<br class="">hawkular-dev mailing list<br class=""><a href="mailto:hawkular-dev@lists.jboss.org" class="">hawkular-dev@lists.jboss.org</a><br class="">https://lists.jboss.org/mailman/listinfo/hawkular-dev<br class=""><br class=""></blockquote><br class="">_______________________________________________<br class="">hawkular-dev mailing list<br class=""><a href="mailto:hawkular-dev@lists.jboss.org" class="">hawkular-dev@lists.jboss.org</a><br class="">https://lists.jboss.org/mailman/listinfo/hawkular-dev<br class=""><br class="">_______________________________________________<br class="">hawkular-dev mailing list<br class="">hawkular-dev@lists.jboss.org<br class="">https://lists.jboss.org/mailman/listinfo/hawkular-dev<br class=""><br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">hawkular-dev mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:hawkular-dev@lists.jboss.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">hawkular-dev@lists.jboss.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="https://lists.jboss.org/mailman/listinfo/hawkular-dev" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">https://lists.jboss.org/mailman/listinfo/hawkular-dev</a></div></blockquote></div><br class=""></body></html>