Grafana Heatmap of Prometheus bucket metrics

Bucketed metrics are useful to see which bucket your numbers fall into. Latency is a common example of a bucketed metric, where you may have a bucket for < 10ms, another for < 50ms, etc. The code to create the metric (assuming opencensus) would look something like this.

String name = "latency";
MeasureLong measure = MeasureLong.create(name, description, unit);
BucketBoundaries buckets = BucketBoundaries.create(ImmutableList.of(10d, 50d, ...);
Aggregation distribution = Aggregation.Distribution.create(buckets);
io.opencensus.stats.Stats.getViewManager()
        .registerView(View.create(View.Name.create(name), description, measure, distribution, tagKeys));

Then to add a new value to the metric

Stats.recorder()
    .tag(tagKey, tagValue)
    .record(measure, 8372d);

This will generate metrics like this (probably at http://localhost:9090/metrics)

latency_bucket{le=0.001, tag=””} 1.0
latency_bucket{le=0.01, tag=””} 2.0
latency_bucket{le=0.1, tag=””} 4.0

Then you can generate a Heatmap of that data like this

To do this

1. create a Heatmap in grafana
2. Under Data source configuration
   1. Your promQL should look something like this:
      sum(rate(latency_bucket{tag=””}[5m])) by (le)
   2. Legend should be “{{le}}
   3. Make sure Format is Heatmap
3. Under Axes configuration
   1. Set Data format to “Time series buckets” (VERY IMPORTANT!!!)

The above heatmap is over time. You can also create a snapshot of the distribution at the current moment like this

Same as above, but promQL would be

100 *sum(rate(latency_bucket{tag=””}[5m])) by (le)
/ ignoring (le) group_left
sum(rate(latency_bucket{le=”+Inf”}[5m]))

Monitoring your App

Use Spring Actuator + Prometheus + Grafana to monitor your java app.

Setup Tutorial

Create custom Metrics