Re: [Hawkular-dev] scope of the agent design

I think we need to consider having the following types of agents * embedded, in-process * co-located in separate process * agent-less where we do management/monitoring from the server side There are different scenarios in which each of the above have advantages. For (only) collecting metrics, an embedded collector would be best as it involves the least overhead. Based on our experience of managing Cassandra in RHQ, I think that there are a lot of situations where agent-less makes things much easier. Consider running repair. It is a cluster-wide operation. In RHQ, we go through a complex workflow of server to agent to Cassandra node to agent to server to next agent to next Cassandra node to agent to server and so on. Managing the repair directly from server to Cassandra would make things a lot easier. With respect to monitoring, the agent, particularly one that runs co-located in a separate process, should be capable of being more than just a collector. I am seeing that more and more with other systems I am looking at. With an Open TSDB extension for example, the agent/client side piece can convert data into a blob format that is optimized for storage. This blob conversion can be done on the server side but it causes a hit on performance. I believe that the DataDog agent is capable of performing client-side aggregations, like computing percentiles. This could be efficient and useful for computing method execution times, response times, etc. With respect to alerting, I think it is something we should consider for an agent that runs its own process. The agent could push alerts onto the bus for instance. And if we look more closely at ISPN, we could do the same with distributed caches. The agent fires an alert by putting an entry into its alert cache to which the server is subscribed for notifications. This could also be used for the agent to obtain alert definitions. Something to consider...