If you're running an Elasticsearch cluster you might want to keep track of the metrics it produces. In this article I explain how you can aggregate the metrics of an Elasticsearch cluster and submit them to Graphite.
You need to have following software installed:
Main software (installable from Pypi)
Installing MetricFactory will install the Wishbone framework as a dependency.
Once installed you should have the metricfactory executable available. You can test this by issuing following command:
$ metricfactory list --group metricfactory.decode
Elasticsearch has a HTTP based API which allows us to poll metrics. The available metrics resources are:
Polling these resources returns a JSON formatted document containing metrics of different Elasticsearch parts.
The wishbone.input.httpclient module allows us to poll and collect these resources.
Decode Elasticsearch metrics
The returned JSON document containing metrics has to be converted into the Metricfactory standard metric format. Once converted they can be processed by other Metricfactory modules. We will convert the metrics into the Graphite format using the Graphite builtin module of Wishbone.
Metricfactory requires a bootstrap file which defines the functionality and eventflow:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37
--- modules: httprequest: module: wishbone.input.httpclient arguments: url: - http://elasticsearch-node-001:9200/_cluster/stats - http://elasticsearch-node-001:9200/_cluster/health - http://elasticsearch-node-001:9200/_nodes/stats - http://elasticsearch-node-001:9200/_stats interval: 1 decode: module: metricfactory.decode.elasticsearch arguments: source: lhi encode: module: wishbone.encode.graphite arguments: prefix: application.elasticsearch. script: false output_screen: module: wishbone.output.stdout output_tcp: module: wishbone.output.tcp arguments: host: localhost port: 2013 routingtable: - httprequest.outbox -> decode.inbox - decode.outbox -> encode.inbox - encode.outbox -> output_tcp.inbox ...
Lets run over the different sections of this bootstrap file.
The routingtable (line 33) determines how modules are connected to each other and therefor determine the flow of events.
The httprequest module instance poll the urls (line 7, 8, 9, 10) which return the available metrics in JSON format. The resources are requested with an interval of 1 second (line 11).
The results coming out this input module then flows into into the decode module (line 13) in which the JSON formatted data is converted to the generic metric format. The decode instance is initialized using the source argument (line 16) which allows you to add the cluster name to the metric names in case you're collecting metrics from multiple cluster instances.
The decoded events are then converted into the required Graphite format by the encode module instance (line 18). The prefix argument (line 21) allows you to define the top scope of the metric names.
Events then go to the output_tcp module which submits the metrics into Graphite itself.
If you first want to experiment with the metric name formatting, you can write the metrics to STDOUT by connecting encode.outbox to output_screen.inbox (line 36).
To start the server, save the above bootstrap configuration to a file and execute following command:
$ metricfactory debug --config bootstrap.yaml