Submit Elasticsearch metrics to Graphite

Prerequisites

You need to have following software installed:

Main software (installable from Pypi)

• Metricfactory

Installing MetricFactory will install the Wishbone framework as a dependency.

Once installed you should have the metricfactory executable available. You can test this by issuing following command:

$ metricfactory list --group metricfactory.decode

Elasticsearch

Elasticsearch has a HTTP based API which allows us to poll metrics. The available metrics resources are:

• Cluster stats
• Nodes stats
• Indices stats

Polling these resources returns a JSON formatted document containing metrics of different Elasticsearch parts.

The wishbone.input.httpclient module allows us to poll and collect these resources.

Decode Elasticsearch metrics

The returned JSON document containing metrics has to be converted into the Metricfactory standard metric format. Once converted they can be processed by other Metricfactory modules. We will convert the metrics into the Graphite format using the Graphite builtin module of Wishbone.

Bootstrapfile

Metricfactory requires a bootstrap file which defines the functionality and eventflow:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

---
modules:
  httprequest:
    module: wishbone.input.httpclient
    arguments:
      url:
        - http://elasticsearch-node-001:9200/_cluster/stats
        - http://elasticsearch-node-001:9200/_cluster/health
        - http://elasticsearch-node-001:9200/_nodes/stats
        - http://elasticsearch-node-001:9200/_stats
      interval: 1

  decode:
    module: metricfactory.decode.elasticsearch
    arguments:
      source: lhi

  encode:
    module: wishbone.encode.graphite
    arguments:
      prefix: application.elasticsearch.
      script: false

  output_screen:
    module: wishbone.output.stdout

  output_tcp:
    module: wishbone.output.tcp
    arguments:
      host: localhost
      port: 2013

routingtable:
  - httprequest.outbox  -> decode.inbox
  - decode.outbox       -> encode.inbox
  - encode.outbox       -> output_tcp.inbox
...

Lets run over the different sections of this bootstrap file.

The routingtable (line 33) determines how modules are connected to each other and therefor determine the flow of events.

The httprequest module instance poll the urls (line 7, 8, 9, 10) which return the available metrics in JSON format. The resources are requested with an interval of 1 second (line 11).

The results coming out this input module then flows into into the decode module (line 13) in which the JSON formatted data is converted to the generic metric format. The decode instance is initialized using the source argument (line 16) which allows you to add the cluster name to the metric names in case you're collecting metrics from multiple cluster instances.

The decoded events are then converted into the required Graphite format by the encode module instance (line 18). The prefix argument (line 21) allows you to define the top scope of the metric names.

Events then go to the output_tcp module which submits the metrics into Graphite itself.

If you first want to experiment with the metric name formatting, you can write the metrics to STDOUT by connecting encode.outbox to output_screen.inbox (line 36).

To start the server, save the above bootstrap configuration to a file and execute following command:

$ metricfactory debug --config bootstrap.yaml

This article has been updated.