Demo: https://youtu.be/huDuOhGcta0

Fluentd and Logstash are two similar tools that can collect data from multiple sources, apply some filtering/processing/routing and forward it to one or more data stores. The company behind Logstash is Elastic of Elasticsearch fame. The company behind fluentd is Treasure data. Logstash is obviously well integrated with Elasticsearch but can by itself not talk to Prometheus, not natively and not through a plugin. Fluentd has a Prometheus output plugin (https://github.com/fluent/fluent-plugin-prometheus) and also one for Elasticsearch (https://github.com/uken/fluent-plugin-elasticsearch).

I think that Prometheus is useful for Mobile App Metrics: some of the data is time-series (e.g. SDK Version usage) and some of the data (e.g. Logs) is better suited for Elasticsearch. Fluentd can cover both scenarios, that's why I focused mostly on it.

The proposed architectures for Mobile App Metrics all rely on a 'Gateway' service where Clients can push their data. I was able to use fluentd for this purpose by extending the metrics-apb (see the Demo video, changes: https://github.com/pb82/metrics-apb/commit/8db9c320f1dfceb57c501562c1e2b6f711152ea2). One could argue that if we have to use a gateway service anyway, why not use an existing, tested one like fluentd instead of writing our own. However there are a few drawbacks to fluentd that I would like to point out:

The Good

• Can talk to Prometheus, Elasticsearch and many more
• No Client libraries needed, clients just use http post
• (claimed) 30-40 Mb of memory usage and 13.000 messages per second per core

The Bad

• Does not play nice with Openshift/Kubernetes: fluentd wants to run as root inside it's container which is not allowed by default. I had to use a service account with special permissions (anyuid). I also tried to modify the Dockerfile so that it works a non-root but was not successful.

The Ugly

• Plugins are written in Ruby and can contain native extensions (this is the case for the Prometheus plugin). Having experienced native Ruby extensions in fhcap, i'm a bit wary.

One question i have after the investigation: are the Metrics static or do we want to allow users to define their own Metrics? Because that would involve changing the fluentd configuration.