Django App Monitoring with Prometheus and Grafana
Monitoring is crucial for ensuring the stability, performance, and availability of your Django applications. In this article, we’ll explore how to set up monitoring using Prometheus and Grafana, two powerful tools that provide deep insights into the behavior of your application. We will also use Docker Compose for more efficient deployment, but you may choose to install both of the tools manually.
Why Monitoring Matters
Before diving into the technical details, let’s understand why monitoring is essential. Monitoring helps you:
- Detect Issues Early: Identify potential problems before they escalate into critical issues.
- Optimize Performance: Analyze performance metrics to fine-tune your application for optimal efficiency.
- Ensure Reliability: Monitor uptime, response times, and error rates to ensure a reliable user experience.
- Capacity Planning: Understand resource usage trends to plan for scaling and resource allocation.
Prometheus
Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. It is specifically built for monitoring cloud-native applications and microservices architectures. Here are key aspects of Prometheus:
- Metrics Collection: Prometheus collects metrics from targets by scraping HTTP endpoints exposed by the targets. These endpoints provide data in a format that Prometheus understands, typically in a simple text-based format.
- Data Model: Prometheus has a powerful data model based on time series data. It stores metrics as time series, where each series consists of a metric name and a set of key-value pairs called labels. This data model allows for flexible querying and analysis of metrics.
- PromQL: Prometheus Query Language (PromQL) is used to query and manipulate metrics data. It provides operators for aggregation, filtering, and mathematical operations, enabling advanced analysis of time series data.
- Alerting: Prometheus includes a robust alerting system. Users can define alert rules based on PromQL expressions to trigger alerts when certain conditions are met. Alerts can be sent via various notification channels such as email, Slack, or custom integrations.
- Service Discovery: Prometheus supports service discovery mechanisms to dynamically discover and monitor targets in a cloud-native environment. It can discover targets using DNS, Kubernetes, EC2, and other service discovery methods.
Grafana
Grafana is a leading open-source platform for monitoring and observability. It provides rich visualization, analytics, and alerting capabilities, making it a popular choice for creating dashboards and analyzing metrics data. Here are key aspects of Grafana:
- Data Source Integration: Grafana can integrate with various data sources, including Prometheus, to visualize metrics data. It supports a wide range of data sources such as databases, time series databases, cloud monitoring services, and more.
- Dashboard Creation: Grafana allows users to create interactive and customizable dashboards using a drag-and-drop interface. Dashboards can include various types of visualizations such as graphs, tables, heatmaps, and gauges.
- Community Driven: Grafana’s development is heavily influenced by its community. Users contribute plugins, dashboards, and improvements, fostering innovation and ensuring that Grafana evolves to meet the dynamic needs of modern monitoring and observability practices.
- Alerting and Notifications: Grafana provides robust alerting capabilities. Users can define alert rules based on metrics data and receive notifications via email, Slack, PagerDuty, or other channels when alerts are triggered.
- Templating and Variables: Grafana supports templating and variables, allowing users to create dynamic dashboards that can adapt to different environments or configurations. Variables can be used to filter data, switch between data sources, or customize dashboard views.
- Data Exploration: Grafana includes powerful data exploration features that enable users to drill down into metrics data, zoom in on specific time ranges, apply filters, and perform ad-hoc analysis directly within the dashboard interface.
Setting Up Prometheus & Grafana
Setting Up django-prometheus
There is a python library called django-prometheus which helps in exporting Django monitoring metrics for Prometheus. To install it simply run this command:
pip install django-prometheusNext, we need to set some things up in our settings.py, add these lines after your INSTALLED_APPS and MIDDLEWARE variable are declared:
INSTALLED_APPS = [
...
"django_prometheus",
]
MIDDLEWARE = (
["django_prometheus.middleware.PrometheusBeforeMiddleware"]
+ MIDDLEWARE
+ ["django_prometheus.middleware.PrometheusAfterMiddleware"]
)
Great! Now let’s make the metrics exported by django-prometheus accessible by adding these in urls.py:
urlpatterns = [
...
path("prometheus/", include("django_prometheus.urls")),
]
Nice work, we should be able to access our metrics by going to /prometheus/metrics endpoint!
Setting Up Prometheus & Grafana with Docker Compose
Below is the docker compose file used to setup both Prometheus & Grafana. Save it to a file called docker-compose.monitoring.yml.
---
version: '3.8'
networks:
monitoring:
driver: bridge
volumes:
prometheus_data: {}
grafana-data: {}
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
restart: unless-stopped
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--web.enable-lifecycle'
ports:
- 9090:9090
networks:
- monitoring
grafana:
image: grafana/grafana-oss:latest
container_name: grafana
ports:
- '3000:3000'
volumes:
- grafana-data:/var/lib/grafana
restart: unless-stopped
networks:
- monitoringDon’t forget to add your Prometheus config file, save it to a file called prometheus.yml. Here is my config for our application.
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'django_app'
metrics_path: '/prometheus/metrics'
static_configs:
- targets: ['host.docker.internal:8080']Basically, above configuration is targeting localhost:8080/prometheus/metrics of the host machine from our Prometheus docker container every 15 seconds.
Now, run below command in your CLI to run the docker compose file and build the images:
docker-compose -f docker-compose.monitoring.yml up --build
We can now access our Grafana & Prometheus by going to localhost:3000 and localhost:9090 respectively.
When accessing Grafana endpoint for the first time, you will be greeted with a login page, input admin as the username and password, and you are in! Furthermore, you will be prompted to change your password and you should definitely do that. Below is the look of Grafana Dashboard after you have successfully logged in as admin.
Adding a Grafana Dashboard
In this section, we will explore how to add a dashboard to monitor our application’s metrics. Remember when I said Grafana is community driven? That means we can add dashboards made by the community! I found a beautiful dashboard which works well with django-prometheus made by andreynovikov (https://grafana.com/grafana/dashboards/17658-django/). Let us now explore how to add that dashboard in our Grafana!
Adding Our Data Source
Remember that our data source is from Prometheus. Let’s add it!
- Go to Data sources
- Click Add data source
- Choose Prometheus, and now our first data source is created.😁
- Scroll down into Connection, and fill Prometheus server URL with http://prometheus:9090, which is our Prometheus docker container’s domain when accessed by our Grafana docker container.
- Scroll to the bottom and click Save & Test, if there are green checkmarks (which hopefully you have) then you are good to go!
Adding The Dashboard
Now, we will add that dashboard made by andreynovikov into our Grafana. Let’s dive right in!
- Go to Dashboards
- Click Create Dashboard
- Click Import dashboard, and fill the “Find and import dashboards” input field with the URL of your desired dashboard, then click Load
- Choose our just created prometheus data source as the data source, and then click Import.
- And, voila! We will be greeted with our imported dashboard.
And there you have it! You’ll now have a comprehensive dashboard showcasing metrics relevant to monitoring your Django app. In our imported dashboard, we are able to see how many requests/seconds, the number of DB query errors, number of requests by processing time, request latency, response statuses, and many more! Those information are useful when you are doing performance testing / stress testing with Apache JMeter. Please see my article about JMeter if you haven’t heard about it: (https://medium.com/@fadhilahazzah04/enhancing-software-quality-through-comprehensive-testing-stress-tests-and-penetration-tests-356a8d59e54b).
If you think that the provided information is not enough, you may add more components by clicking Add button in the bar below the navigation bar.
Alternatively, you could search for another dashboard more tailored to your needs. You can explore more dashboards made by the community just by visiting https://grafana.com/grafana/dashboards/.
Conclusion
Monitoring your Django app with Prometheus and Grafana using Docker Compose provides a robust solution for gaining insights into your application’s performance and reliability. By leveraging these powerful tools and the Docker Compose environment, you can streamline the setup process and focus on optimizing your application.
Continuous monitoring allows you to detect issues early, optimize performance, ensure reliability, and plan for scaling as your application grows. With Prometheus for metrics collection, Grafana for visualization and analysis, and Docker Compose for efficient deployment, you have a powerful monitoring stack at your disposal.
In conclusion, monitoring is not just about reacting to problems but also about proactively managing and improving your application’s performance. With the tools and techniques discussed in this article, you can take your monitoring capabilities to the next level and ensure a seamless user experience for your application.
