Monitoring Docker Container Performance with Prometheus and Grafana

Kamalesh D

Published in

DevOps.dev

9 min readJul 26, 2023

New day, New Topic…. Let’s learn along..!!

Grafana Vizualization Dashboard:

A dashboard provides a comprehensive overview of your data, enabling you to monitor metrics using various visualizations. Dashboards are composed of panels, each contributing to the narrative you want to present. Every panel comprises a query, specifying the data you wish to showcase, and a visualization, determining how the data is visually represented.

What powers your monitoring and keeps a vigilant eye on your systems? Meet Prometheus, the vigilant guardian of your infrastructure! 🚀

Prometheus, the open-source monitoring and alerting system, is tailored for DevOps practices. Employing a pull-based model, it gathers metrics from diverse sources such as servers, containers, and applications. These metrics are then stored in a time-series database, enabling robust querying and visualization options. With its flexible alerting mechanism based on predefined thresholds, Prometheus seamlessly integrates with popular notification tools, ensuring timely and proactive responses to potential issues.

Set up Docker and initiate the Docker service on a Linux EC2 instance using USER DATA.

Launch a new EC2 instance and select the “Ubuntu Server 22.04 LTS” as the Amazon Machine Image (AMI).
Choose the “t2.micro” instance type.
Proceed to the “Configure Instance Details” page during the setup.
In the “Advanced Details” section, you will find the “User data” field.
Copy and paste the script provided below into the “User data” field.

#!/bin/bash

# Update the system
apt-get update
apt-get upgrade -y

# Install Docker dependencies
apt-get install -y apt-transport-https ca-certificates curl software-properties-common

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Add Docker repository
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Update package information and install Docker
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io

# Start Docker service
systemctl start docker

# Enable Docker to start on system boot
systemctl enable docker

Deploying Two Docker Containers Running a Basic Application (Todo App)

Step- 01

First we have to install Grafana-server for monitoring purpose. You can follow this official installation guide here or below commands:

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

Add this repository for stable releases:

echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Add this repository if you want beta releases:

echo "deb https://packages.grafana.com/oss/deb beta main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

After you add the repository:

sudo apt-get update
sudo apt-get install grafana

Step- 02

Follow this blog to run the simple node.js docker container app CLICK HERE..
Follow below steps to run 2nd container application.
we are going to run apache2 server on same ec2 instance.

#Create a new file called "Dockerfile" in a directory of your choice.
$ touch Dockerfile
#Open the Dockerfile using a text editor of your choice.
$ nano Dockerfile
#Add the following content to the Dockerfile:
FROM httpd:2.4
COPY index.html /usr/local/apache2/htdocs/
EXPOSE 80
#Save and exit the Dockerfile.
#Create a new file called "index.html" in the same directory as the Dockerfile.
$ touch index.html
#Open the index.html file using a text editor.
$ nano index.html
#Add the following content to the index.html file:
<!DOCTYPE html>
<html>
<head>
    <title>Hello, World!</title>
</head>
<body>
    <h1>Hello, World!</h1>
</body>
</html>
#Save and exit the index.html file.
#Build the Docker image using the following command:
$ docker build -t docker_grafana.
#Run the Docker container using the following command:
$ docker run -d -p 8080:80 docker_grafana-image

I have installed above mentioned all tools here is Screeen shot.

Step-3

Run Prometheus monitoring tool using docker container. Follow below steps
Create file prometheus.yaml write below mentioned configuration code of promethues in that file.

#create file
sudo vi prometheus.yaml
#write configuations in this file
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:9090']
#save this file

After successfully edit the prometheus.yaml file you have to run docker container for above configurations.
Run folllowing command:

sudo docker run -d --name prometheus -v $(pwd):/etc/config -p 9090:9090 prom/prometheus --config.file=/etc/config/prometheus.yaml

Let’s break down the Docker run command and explain each part:

docker run: This is the main command to create and start a Docker container.
-d: This option runs the container in the background or detached mode, allowing it to continue running even after the terminal session is closed.
--name prometheus: With this option, we assign the name "prometheus" to the container, making it easier to reference and manage.
-v $(pwd):/etc/config: This part mounts the current working directory ($(pwd)) on the host machine as a volume inside the container at the path /etc/config. This enables the container to access the Prometheus configuration file present in the current directory.
-p 9090:9090: Here, we map port 9090 of the host machine to port 9090 of the container. This port mapping is essential to access the Prometheus web interface running inside the container.
prom/prometheus: It specifies the Docker image that will be used to create the container. In this case, the command uses the official Prometheus image from the Docker Hub.
--config.file=/etc/config/prometheus.yaml: This part sets the configuration file for Prometheus inside the container to /etc/config/prometheus.yaml. It assumes that a prometheus.yaml configuration file is present in the current directory and mounted as a volume.

After running the command, you can use sudo docker ps to view the running containers and ensure that the Prometheus container is up and running.

To allow inbound traffic for the required ports in the AWS console security group, follow these steps:

Open the AWS Management Console and navigate to the EC2 dashboard.
Locate and select the EC2 instance that is running the Docker containers.
Click on the “Security Groups” tab in the instance details.
Select the security group associated with the EC2 instance.
Click on the “Inbound Rules” tab to manage inbound traffic.
Add the following inbound port rules:

Port 9090: Allow inbound traffic for Prometheus.
Port 49160: Allow inbound traffic for the Node.js app server.
Port 8080: Allow inbound traffic for the Apache server.
Port 3000: Allow inbound traffic for Grafana.
Port 9323: Allow inbound traffic for Docker logs.

For each port rule, choose the appropriate source (e.g., specific IP, CIDR block, or “Anywhere” for open access).
Save the changes to apply the new inbound rules to the security group.

Now, the required ports are open in the security group, allowing external access to the corresponding services running in the Docker containers on your EC2 instance.

After all steps completion you can get prometheus dashboard like this on http://<public-ip>:9090

Integrate the Docker containers and establish real-time log sharing with Grafana.

Sure, here are the instructions in points:

Locate the configuration file of the Docker daemon, daemon.json. The file is usually found in the /etc/docker/ directory.
If the daemon.json file is not present in the specified location, create a new file named daemon.json.
Open the daemon.json file using a text editor.
Add the following code snippet to the daemon.json file:

{
        "metrics-addr" : "0.0.0.0:9323",
        "experimental" : true
}

#if this code not working then you can try following code 

{
        "metrics-addr" : "127.0.0.1:9323",
        "experimental" : true
}

For apply this change you have to restart docker service sudo systemctl restart docker
The containers will be paused state we need to start running it type the below command to start it

docker start <container id>

Now wait some time and then check logs on http://<public-ip>:9323/metrics
You should get docker logs on webpage like following.

Now we have to add new target in prometheus.yaml file as follows.
Add Docker job in prometheus.yaml file.

- job_name: 'docker'
  scrape_interval: 5s
  static_configs:
    - targets: ['<docker_logs IP>:9323']

After above step you have to restart Prometheus docker container docker restart <container id>`
The Target of docker will added in prometheus targets section.

Check the logs and create docker dashboard on Grafana UI.

Go to CLI and type the below command to start Grafana server.

sudo grafana server

To create Dashboard in Grafana you have to login with admin admin user and password on http://<public-ip>:3000
After login click on Add your first data source.

Then select Prometheus as Data source.
Now you have to put prometheus server url like as shown below.

After integrating Docker containers and Grafana, click on “Save & Test.” A “Successfully done” message dialog will appear, confirming the setup is working correctly.
In Grafana, you have the flexibility to create various dashboards to suit your requirements. For this tutorial, we will create a dashboard with four panels.
To get started, create the first panel that displays the number of Engine CPU.
Click on “Build Dashboard” and then select “New Panel” to add a new panel to the dashboard.
In the “Metrics browser” selection, choose engine_daemmon_engine_cpus_cpus.
Select the “Job” and set the job name as “docker.”
Click on “Run queries,” and Grafana will fetch the number of Engine CPU metrics.
Now, repeat the process to add the second panel, which will display the number of running containers.
Add the third panel to show the count of paused containers.
Lastly, create the fourth panel to display the number of stopped containers.
Once all four panels are added, arrange and customize them on the dashboard as per your preference.

By following these steps, you will have a Grafana dashboard with four panels showing different metrics related to the Docker containers, providing you with valuable insights into their performance and status.

In the Metrics browser selection select engine_daemmon_container_states_containers` then select State filter and state is Pause. Then click Run queries will fetch the number of Paused Container as shown below.
In the Metrics browser selection select engine_daemmon_container_states_containers` then select State filter and state is stopped . Then click Run queries will fetch the number of Stopped Container as shown below.

That’s all about today’s task of DevOps journey

Thankyou for reading 👍

If you liked this story then click on 👏👏 do follow for more interesting and helpful stories.