Install RabbitMQ

I followed the instructions provided on the docker hub page here. There are two tags available, I chose the one which had the management plugin pre-configured. This management plugin provides a basic but very helpful management UI for administration. I would recommend having this plugin as default in your RabbitMQ setup but do restrict “HTTP/HTTPS” access to port “15672”. You shouldn’t be exposing RabbitMQ admin console to the outside world.

docker run -d --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.8.16-management

If you are using an older version of RabbitMQ then follow the instructions here to install and configure Prometheus exporter. For this tutorial, I am using the latest 3.8.x version that by default has this plugin.

As of 3.8.0, RabbitMQ ships with built-in Prometheus & Grafana support.

https://www.rabbitmq.com/prometheus.html

# get a prompt to the running container
docker exec -it rabbitmq /bin/bash

# install curl in the docker container
apt-get update
apt-get -y install curl

# verify if rabbitmq_prometheus plugin is installed and enabled -- within container
curl -v -H "Accept:text/plain" "http://localhost:15692/metrics"

Re run the RabbitMQ container but open access for port 15692

docker run -d --name rabbitmq \
-p 5672:5672 -p 15672:15672 -p 15692:15692 \
rabbitmq:3.8.16-management

Configure Prometheus to scrape RabbitMQ

Now add this exporter into Prometheus as a target. Edit prometheus.yml file in your Prometheus server directory and add the following entry under scrape_configs. Replace localhost with your RabbitMQ server name.

scrape_configs:
- job_name: rabbitmq
  scrape_interval: 60s
  static_configs:
  - targets:
    - localhost:15692

Restart Prometheus service and open Prometheus UI by navigating to http://<hostname&gt;:9090, go to “Status” -> “Targets”, it should show 2 targets now.

Publish and Consume sample messages

Now that we have RabbitMQ running and configured Prometheus to scrape it, let us create a queue and send some messages before we move to Grafana.

To create Queue, navigate to RabbitMQ management UI at http://localhost:15672. For this tutorial, I created a queue of type classic using the below settings.

Though we can publish and receive messages using management UI, this approach is better suited for queue configuration testing. For this tutorial, I am using the following Python code to publish and consume messages. For full documentation of pika library please see this link.

Producer for RabbitMQ

''' 
Simple Producer which generates 500 test messages with random delay
'''
import pika
import json
import random
from faker import Faker
from time import sleep

fake = Faker()

connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

# persist messages in case of failure
msg_prop = pika.BasicProperties(
            delivery_mode = 2, # make message persistent
           )

#send 500 messages with random delay 
for i in range(1, 500):
    message = {
        "name": fake.name(),      
        "email_id": fake.email(),      
        "address": fake.address()
        }

    # Queue was created as durable
    channel.basic_publish(exchange='',
                          routing_key='demo_queue', 
                          properties=msg_prop,
                          body=json.dumps(message))    
    # sleep for few random seconds
    sleep(random.randint(1, 9))
        
connection.close()

CONSUMER for RabbitMQ

'''
Simple consumer to read from the queue with random delay and ack
'''
import pika
import json
import random
from time import sleep

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

def callback(ch, method, properties, body):    
    print("Received {}".format(json.loads(body.decode())))
    # random delay
    r = random.randint(10, 30)
    sleep(r)
    
    # send ack for only few random messages
    if r%2 == 0: 
        ch.basic_ack(delivery_tag = method.delivery_tag)

channel.basic_consume(queue='demo_queue',                      
                      on_message_callback=callback)

channel.start_consuming()
connection.close()

Grafana dashboard for RabbitMQ

You can find detailed information on RabbitMQ client support for various languages here. Now that we have a producer and a consumer (you can run multiple consumers for a single queue), let us configure Grafana which will be our Monitoring and Alerting visualization tool.

To help Grafana users, RabbitMQ provides few ready-to-use Grafana dashboards (6 as of now). You can create new dashboards and customize existing ones to suit your application’s business need. First, I will import a dashboard called RabbitMQ-Overview into Grafana, which I think is a must-have for any RabbitMQ environment. The steps are given in detail here.

The charts are self-explanatory and very handy when it comes to the visibility of the queues and other components of RabbitMQ. So far, we have configured monitoring for RabbitMQ server and its components, but what about visibility into the underlying VM/Container/EC2 on which this Queue is running.

Configure cadvisor for Container Metrics

To have better visibility, we must also monitor at least CPU, Memory and Disk space utilization of the underlying compute infrastructure. For this tutorial I am using docker container to run RabbitMQ so will follow this awesome guide to use cadvisor to monitor container metrics.

# Pull and run cadvisor container
docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--name cadvisor \
-p 8080:8080 \
gcr.io/cadvisor/cadvisor

# Update Prometheus config file to scrape the metrics exposed by cadvisor
scrape_configs:
- job_name: cadvisor
  static_configs:
  - targets:
    - localhost:8080

Restart Prometheus and navigate to the “Status” section to see that a new target has been added. This tutorial is using localhost for all the setup but you should replace “localhost” with the appropriate IP or Hostname in the prometheus.yml file.

You can list container metrics with prefix as “container_” and filter based on container name such as shown below. For a full list of cadvisor metrics, please see this doc.

• container_cpu_usage_seconds_total {name=”rabbitmq”}
• container_memory_usage_bytes {name=”rabbitmq”}
• container_memory_max_usage_bytes {name=”rabbitmq”}
• container_fs_usage_bytes {name=”rabbitmq”}
• container_fs_limit_bytes {name=”rabbitmq”}

There are many community-built Grafana dashboards for Prometheus and Docker which you can try or improve upon. I created a sample dashboard to represent 3 metrics for my RabbitMQ container.

Conclusion

I will strongly recommend configuring alerts in Grafana. Some of the important metrics I will suggest being alerted if they cross a certain threshold are as follows

1. Unacknowledged Messages
2. Messages in Queue
3. Messages Delivered
4. Connections open
5. Publishers count
6. Consumers count
7. Disk space utilization
8. CPU utilization
9. Memory utilization

Thanks for reading