How To Install Prometheus on Rocky Linux 9

In this tutorial, we will show you how to install Prometheus on Rocky Linux 9. Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. It’s particularly well-suited for dynamic environments like containerized deployments, microservices architectures, and cloud-native applications. By installing Prometheus on Rocky Linux 9, you’ll gain powerful insights into your system’s performance and health.

Table of Contents

Prerequisites

Before we begin the installation process, ensure you have the following:

A Rocky Linux 9 system with minimal installation
Root or sudo privileges on your system
Basic familiarity with the Linux command line
A stable internet connection for downloading packages

System Preparation

To ensure a smooth installation process, we’ll start by updating our system and configuring some basic settings.

Update System Packages

First, let’s update all existing packages on your Rocky Linux 9 system:

sudo dnf update -y
sudo dnf upgrade -y

After the update process completes, it’s recommended to reboot your system to ensure all changes take effect:

sudo reboot

Configure Firewall Settings

Prometheus uses port 9090 by default. We need to open this port in the firewall to allow incoming connections:

sudo firewall-cmd --permanent --add-port=9090/tcp
sudo firewall-cmd --reload

Create Prometheus User and Directories

For security reasons, it’s best to run Prometheus under a dedicated user account with limited privileges:

sudo useradd --no-create-home --shell /sbin/nologin prometheus

Now, let’s create the necessary directories for Prometheus:

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus

Set Up Directory Permissions

Ensure the Prometheus user has the correct permissions on these directories:

sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Installing Prometheus

With our system prepared, we can now proceed with the Prometheus installation.

Download Latest Prometheus Binary

First, we’ll download the latest Prometheus release. Check the official Prometheus download page for the most recent version. As of this writing, we’ll use version 2.45.1:

cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v3.0.1/prometheus-3.0.1.linux-amd64.tar.gz

Extract and Configure Binary Files

Extract the downloaded archive and move the Prometheus binaries to the appropriate locations:

tar xvf prometheus-3.0.1.linux-amd64.tar.gz
cd prometheus-3.0.1.linux-amd64
sudo mv prometheus promtool /usr/local/bin/

Set Up Configuration Directories

Move the configuration files and libraries to their respective directories:

sudo mv consoles/ console_libraries/ /etc/prometheus/
sudo mv prometheus.yml /etc/prometheus/prometheus.yml

Configure prometheus.yml

The `prometheus.yml` file is the main configuration file for Prometheus. Let’s edit it to suit our needs:

sudo nano /etc/prometheus/prometheus.yml

Here’s a basic configuration to get you started:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

This configuration tells Prometheus to scrape metrics from itself every 15 seconds. You can add more targets as you expand your monitoring setup.

Service Configuration

To ensure Prometheus runs as a service and starts automatically on boot, we need to create a systemd service file.

Create Systemd Service File

Create a new file named `prometheus.service` in the systemd directory:

sudo nano /etc/systemd/system/prometheus.service

Add the following content to the file:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Enable and Start Prometheus Service

Now, let’s enable and start the Prometheus service:

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

Verify Service Status

Check if Prometheus is running correctly:

sudo systemctl status prometheus

You should see output indicating that the service is active and running.

Basic Configuration

With Prometheus installed and running, let’s look at some basic configuration options to get you started.

Configure Targets

To monitor additional targets, you’ll need to add them to the `prometheus.yml` file. For example, to monitor a Node Exporter running on another machine:

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter-ip:9100']

Replace `node-exporter-ip` with the actual IP address of your Node Exporter.

Set Up Scrape Intervals

You can adjust the global scrape interval or set specific intervals for different jobs:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

Basic prometheus.yml Settings

Here are some additional settings you might want to configure in your `prometheus.yml`:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

Security Considerations

While this guide focuses on a basic setup, it’s crucial to consider security when deploying Prometheus in a production environment. Some key points to consider:

Use HTTPS for all communications
Implement authentication for the Prometheus web interface
Restrict network access to Prometheus and its targets
Regularly update Prometheus and all related components

Node Exporter Setup

Node Exporter is a Prometheus exporter for hardware and OS metrics. It’s an essential component for monitoring your Linux systems.

Installation Process

To install Node Exporter:

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar xvf node_exporter-1.8.2.linux-amd64.tar.gz
sudo mv node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/

Configuration Steps

Create a systemd service file for Node Exporter:

sudo nano /etc/systemd/system/node_exporter.service

Add the following content:

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Integration with Prometheus

Add Node Exporter as a target in your `prometheus.yml`:

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

Service Management

Enable and start the Node Exporter service:

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

Verification and Testing

After setting up Prometheus and Node Exporter, it’s important to verify that everything is working correctly.

Access Web Interface

Open a web browser and navigate to `http://your_server_ip:9090`. You should see the Prometheus web interface.

How To Install Prometheus on Rocky Linux 9

Check Targets Status

In the Prometheus web interface, go to Status > Targets. You should see your configured targets (including Prometheus itself and Node Exporter) with the state “UP”.

Verify Metrics Collection

In the Prometheus web interface, click on “Graph” and start typing metric names in the expression box. You should see autocomplete suggestions for various metrics.

Basic Query Testing

Try some basic queries to ensure metrics are being collected:

For CPU usage: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
For memory usage: 100 * (1 - ((node_memory_MemFree_bytes + node_memory_Cached_bytes + node_memory_Buffers_bytes) / node_memory_MemTotal_bytes))

Troubleshooting Guide

If you encounter issues during the installation or configuration process, here are some common problems and their solutions:

Common Installation Issues

Permission denied errors: Ensure you’re using sudo for commands that require elevated privileges.
Package conflicts: Make sure your system is up-to-date before installing Prometheus.

Service Startup Problems

If Prometheus fails to start:

Check the service status: sudo systemctl status prometheus
Review logs: sudo journalctl -u prometheus

Configuration Errors

If Prometheus starts but doesn’t work as expected:

Validate your configuration: promtool check config /etc/prometheus/prometheus.yml
Check for syntax errors in your YAML files

Permission Issues

If Prometheus can’t access certain files or directories:

Verify ownership: ls -l /etc/prometheus /var/lib/prometheus
Adjust permissions if necessary: sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

Best Practices and Security

To ensure the optimal performance and security of your Prometheus installation, consider the following best practices:

Security Recommendations

Use TLS encryption for all Prometheus communications
Implement authentication for the Prometheus web interface
Regularly update Prometheus and all related components
Use a firewall to restrict access to Prometheus ports

Backup Considerations

Regularly backup your Prometheus configuration files
Consider using remote storage for long-term metric storage
Document your setup and configuration for easy recovery

Performance Optimization

Adjust scrape intervals based on your specific needs
Use service discovery for dynamic environments
Implement recording rules for frequently used queries

Monitoring Recommendations

Set up alerting for critical metrics
Use Grafana or similar tools for visualization
Monitor Prometheus itself to ensure it’s functioning correctly

Advanced Configuration

As you become more familiar with Prometheus, you might want to explore some advanced configuration options:

Alert Manager Integration

Alert Manager handles alerts sent by Prometheus server. To integrate it:

Install Alert Manager
Configure Alert Manager in your prometheus.yml
Set up alerting rules in Prometheus

Custom Scrape Configurations

Prometheus supports various service discovery mechanisms. You can use these to dynamically discover and monitor targets in cloud environments or container orchestration platforms.

Additional Exporters

There are numerous exporters available for different systems and applications. Some popular ones include:

MySQL Exporter for database monitoring
Blackbox Exporter for probing endpoints over HTTP, HTTPS, DNS, TCP, and ICMP
cAdvisor for container monitoring

High Availability Setup

For critical environments, you might want to set up Prometheus in a high availability configuration. This typically involves running multiple Prometheus instances and using a load balancer to distribute requests.

Congratulations! You have successfully installed Prometheus. Thanks for using this tutorial for installing the Prometheus monitoring system tool on your Rocky Linux 9 system. For additional help or useful information, we recommend you check the official Prometheus website.

VPS Manage Service Offer

If you don’t have time to do all of this stuff, or if this is not your area of expertise, we offer a service to do “VPS Manage Service Offer”, starting from $10 (Paypal payment). Please contact us to get the best deal!