How To Install Prometheus on CentOS Stream 10

Prometheus has become an essential tool for monitoring infrastructure and applications in modern environments. As an open-source monitoring and alerting toolkit, it provides powerful capabilities for collecting, storing, and analyzing metrics from your systems. In this comprehensive guide, we’ll walk through the complete process of installing and configuring Prometheus on CentOS Stream 10. Whether you’re managing a small server or a large-scale deployment, this step-by-step tutorial will help you set up a robust monitoring solution.

What is Prometheus?

Prometheus is a powerful open-source monitoring and alerting system designed for reliability and scalability. Originally developed at SoundCloud in 2012, it has since become a standalone project and was the second project to join the Cloud Native Computing Foundation (CNCF) after Kubernetes.

Prometheus works by collecting metrics from configured targets at specified intervals, storing the data locally, and running rules over this data to aggregate and record new time series or generate alerts. The system features a multi-dimensional data model where time series data is identified by metric name and key-value pairs, a flexible query language (PromQL), and an efficient time series database.

Key features that make Prometheus stand out include:

• Pull-based metrics collection
• No reliance on distributed storage
• A powerful query language for data analysis
• Autonomous server operation
• Service discovery integration
• Visualization capabilities through a built-in expression browser
• Support for multiple client libraries and exporters

Unlike traditional monitoring tools that rely on agent-based push mechanisms, Prometheus actively scrapes metrics endpoints, giving you more control over the monitoring process. This makes it particularly well-suited for dynamic container environments and microservices architectures commonly deployed on CentOS Stream 10.

Prerequisites

Before installing Prometheus on your CentOS Stream 10 system, ensure you have the following prerequisites in place:

System Requirements:

• CentOS Stream 10 (minimal installation is sufficient)
• At least 2GB RAM (4GB recommended for production environments)
• 2 CPU cores (minimum)
• 20GB of free disk space (more for extended retention periods)
• Internet connectivity for downloading packages

User Permissions:

• You’ll need a user with sudo privileges or root access
• Ability to create system users and groups

Required Packages:

• wget or curl for downloading files
• tar for extracting compressed files
• Basic text editors (vim, nano, etc.)

Network Requirements:

• Port 9090 must be accessible for the Prometheus web interface
• Firewall rules allowing traffic to monitoring ports
• Optional: internet access for remote monitoring endpoints

Update your system packages before proceeding to ensure you’re working with the latest security patches and dependencies:

sudo dnf update -y
sudo dnf install wget tar -y

This preparation ensures your CentOS Stream 10 system is ready for a smooth Prometheus installation process.

Preparing Your CentOS Stream 10 System

Before installing Prometheus, we need to prepare our CentOS Stream 10 environment properly. This includes updating packages, creating the necessary user accounts, and setting up the required directory structure.

Updating System Packages

First, let’s ensure our system is up-to-date:

sudo dnf update -y
sudo dnf install wget -y

A system reboot may be necessary after a significant update:

sudo reboot

Creating a Prometheus User

For security purposes, we’ll create a dedicated system user for running Prometheus. This user will have restricted privileges and no home directory:

sudo useradd --no-create-home --shell /bin/false prometheus

This approach follows the principle of least privilege, ensuring that the Prometheus service runs with only the permissions it needs.

Setting Up Directory Structure

Now, let’s create the necessary directories for Prometheus configuration and data storage:

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus

The /etc/prometheus directory will store configuration files, while /var/lib/prometheus will contain the time-series database where Prometheus stores metrics data.

Let’s set the correct ownership for these directories:

sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Configuring Firewall Rules

If you have a firewall enabled (which is recommended), you’ll need to allow traffic to port 9090, which is the default port for the Prometheus web interface:

sudo firewall-cmd --permanent --add-port=9090/tcp
sudo firewall-cmd --reload

This configuration ensures that you can access the Prometheus web interface from other machines in your network.

Downloading and Installing Prometheus

With our system prepared, we can now download and install Prometheus. This process involves downloading the binary release, extracting it, and moving the files to their appropriate locations.

Finding the Latest Release

Visit the official Prometheus downloads page or check the GitHub releases page to find the latest version. For our example, we’ll use version 2.45.1, but you should replace this with the latest stable release available.

Downloading Prometheus

Let’s download the Prometheus binary package:

cd /opt
sudo wget https://github.com/prometheus/prometheus/releases/download/v3.2.1/prometheus-3.2.1.linux-amd64.tar.gz

Verifying the Download

It’s good practice to verify the integrity of the downloaded file:

sudo sha256sum prometheus-3.2.1.linux-amd64.tar.gz

Compare the output with the checksum provided on the Prometheus download page to ensure the file hasn’t been tampered with.

Extracting Prometheus

Extract the downloaded tarball:

sudo tar -xvzf prometheus-3.2.1.linux-amd64.tar.gz

For easier reference, rename the extracted directory:

sudo mv prometheus-3.2.1.linux-amd64.tar.gz prometheus

Moving Files to Their Proper Locations

Now, let’s organize the Prometheus files into their appropriate directories:

# Copy binaries
sudo cp prometheus/prometheus /usr/local/bin/
sudo cp prometheus/promtool /usr/local/bin/

# Copy console libraries and templates
sudo cp -r prometheus/consoles /etc/prometheus/
sudo cp -r prometheus/console_libraries /etc/prometheus/

# Copy the example configuration
sudo cp prometheus/prometheus.yml /etc/prometheus/

Setting Correct Permissions

Set the correct ownership for all the files:

sudo chown -R prometheus:prometheus /usr/local/bin/prometheus
sudo chown -R prometheus:prometheus /usr/local/bin/promtool
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
sudo chown -R prometheus:prometheus /etc/prometheus/prometheus.yml

This ensures that the Prometheus user can access and modify all the necessary files.

Directory Structure Setup

Understanding the Prometheus directory structure is essential for proper maintenance and configuration. Here’s a breakdown of the key directories and their purposes:

/etc/prometheus

This directory contains all configuration files for Prometheus:

• prometheus.yml: The main configuration file that defines global settings, scrape configurations, and rule files
• consoles/: Contains web console template files
• console_libraries/: Contains JavaScript libraries for the web console
• rules/: Optional directory for storing alerting and recording rules

/var/lib/prometheus

This directory stores the time-series database (TSDB) data. Prometheus will write all collected metrics to this location. The size of this directory will grow over time depending on:

• The number of metrics you collect
• Your retention period settings
• The scrape interval configuration

/usr/local/bin

This directory contains the Prometheus binaries:

• prometheus: The main Prometheus server binary
• promtool: A utility for tasks like checking configuration and rule files

Ensure all directories have the correct permissions to prevent access issues:

sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown -R prometheus:prometheus /var/lib/prometheus
sudo chmod -R 775 /etc/prometheus
sudo chmod -R 775 /var/lib/prometheus

This directory structure follows the Filesystem Hierarchy Standard (FHS) convention used in Linux systems, making your Prometheus installation organized and maintainable.

Basic Prometheus Configuration

The Prometheus configuration file (prometheus.yml) controls how Prometheus collects and manages metrics. Let’s create a basic configuration that will work for most environments.

Understanding the Configuration File

The default prometheus.yml file contains several sections:

• global: Sets default parameters like scrape intervals and evaluation periods
• alerting: Configures how Prometheus connects to Alertmanager instances
• rule_files: Lists files containing recording and alerting rules
• scrape_configs: Defines what targets Prometheus will monitor

Creating a Basic Configuration

Edit the Prometheus configuration file:

sudo nano /etc/prometheus/prometheus.yml

Replace its contents with this basic configuration:

# Global settings
global:
  scrape_interval: 15s     # How frequently to scrape targets
  evaluation_interval: 15s # How frequently to evaluate rules

# Rule files (empty for now)
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# Scrape configurations
scrape_configs:
  # Self-monitoring
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  
  # CentOS system monitoring (we'll set up Node Exporter later)
  - job_name: 'centos_node'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9100']

Understanding Scrape Targets

In the configuration above:

• The first job monitors Prometheus itself (port 9090)
• The second job is configured to monitor the Node Exporter (port 9100), which we’ll install later

Setting Ownership

Make sure the configuration file has the correct ownership:

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

Validating the Configuration

Before proceeding, verify that your configuration file is valid:

sudo -u prometheus /usr/local/bin/promtool check config /etc/prometheus/prometheus.yml

This command will check for syntax errors and configuration issues before you start the Prometheus service.

Setting Up Prometheus as a Systemd Service

To ensure Prometheus runs reliably and starts automatically with your system, we’ll configure it as a systemd service. This approach provides better management, automatic startup, and proper logging.

Creating the Systemd Service File

Create a new systemd service file:

sudo nano /etc/systemd/system/prometheus.service

Add the following content to the file:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

This configuration:

• Ensures Prometheus starts after network services are available
• Runs Prometheus as the dedicated prometheus user
• Specifies the paths to all necessary files and directories
• Makes Prometheus start automatically on system boot

Loading the Service

After creating the service file, reload the systemd daemon to register the new service:

sudo systemctl daemon-reload

Enabling and Starting Prometheus

Now, enable and start the Prometheus service:

sudo systemctl enable prometheus
sudo systemctl start prometheus

Verifying the Service Status

Check if Prometheus is running correctly:

sudo systemctl status prometheus

You should see an output indicating that the service is active (running). If there are any issues, this command will show error messages that can help with troubleshooting.

Starting and Testing Prometheus

With Prometheus installed and configured as a service, it’s time to verify that everything is working correctly.

Checking the Service Status

First, verify that the Prometheus service is running:

sudo systemctl status prometheus

The output should indicate that the service is “active (running)”. If you see any errors, check the Prometheus logs:

sudo journalctl -u prometheus -f

Accessing the Web Interface

Prometheus comes with a built-in web interface that allows you to query metrics, view targets, and explore the status of your monitoring setup. Access it by opening a web browser and navigating to:

http://your_server_ip:9090

Replace your_server_ip with the actual IP address of your CentOS Stream 10 server.

Exploring the Interface

Once you access the web interface, you’ll see several tabs:

• Graph: For querying and visualizing metrics
• Status: Contains information about the current state of Prometheus
• Targets: Shows all configured monitoring targets and their health
• Alerts: Displays configured alerts (empty by default)
• Help: Provides documentation and assistance

Verifying Metric Collection

To verify that Prometheus is collecting its own metrics:

1. Click on the “Graph” tab
2. In the query box, enter: up
3. Click “Execute”
4. Switch to the “Graph” view to see a visualization

If everything is working correctly, you should see a value of 1 for the up metric for the prometheus job, indicating that the target is up and being scraped successfully.

Setting Up Node Exporter

While Prometheus itself provides valuable metrics about its own operation, to monitor your CentOS Stream 10 system effectively, you’ll need Node Exporter. This tool collects hardware and OS metrics like CPU usage, memory, disk space, and network statistics.

Installing Node Exporter

First, create a system user for Node Exporter:

sudo useradd --no-create-home --shell /bin/false node_exporter

Download the latest Node Exporter binary:

cd /opt
sudo wget https://github.com/prometheus/node_exporter/releases/download/v1.9.0/node_exporter-1.9.0.linux-amd64.tar.gz

Extract the files:

sudo tar -xvzf node_exporter-1.9.0.linux-amd64.tar.gz

Copy the binary to the appropriate location:

sudo cp node_exporter-1.9.0.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Creating a Systemd Service for Node Exporter

Create a systemd service file for Node Exporter:

sudo nano /etc/systemd/system/node_exporter.service

Add the following content:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Reload systemd, then enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

Verify that the service is running:

sudo systemctl status node_exporter

Opening Firewall Port for Node Exporter

If you’re using a firewall, allow access to the Node Exporter port:

sudo firewall-cmd --permanent --add-port=9100/tcp
sudo firewall-cmd --reload

Verifying Node Exporter Metrics

To verify that Node Exporter is providing metrics, access:

http://your_server_ip:9100/metrics

You should see a long list of system metrics being exposed.

Security Considerations

While getting Prometheus up and running is the first step, securing your monitoring infrastructure is equally important, especially in production environments.

Basic Authentication

By default, Prometheus doesn’t include authentication. To add basic authentication:

1. Install apache2-utils to use htpasswd:

   sudo dnf install httpd-tools -y

2. Create a password file:

   sudo htpasswd -c /etc/prometheus/.htpasswd prometheus_admin

3. Update your Prometheus configuration to use a reverse proxy like Nginx or Apache with authentication.

TLS/SSL Configuration

To encrypt communications to your Prometheus instance:

1. Generate or obtain SSL certificates for your server
2. Configure a reverse proxy (Nginx/Apache) with SSL
3. Point the proxy to your Prometheus instance running on localhost

Firewall Configuration

Limit access to your Prometheus instance by configuring firewall rules:

# Allow access only from specific IP addresses
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="trusted_ip_address" port port="9090" protocol="tcp" accept'

# Remove the general access rule if it exists
sudo firewall-cmd --permanent --remove-port=9090/tcp

# Reload the firewall
sudo firewall-cmd --reload

Running with Restricted Privileges

Ensure Prometheus runs with minimal permissions:

1. Verify the prometheus user has no login shell
2. Use read-only file system mounts where possible
3. Consider running Prometheus in a container for additional isolation

Implementing these security measures will help protect your monitoring infrastructure from unauthorized access and potential attacks.

Advanced Configuration Options

As your monitoring needs grow, you might want to extend Prometheus with advanced configurations. Here are some options to enhance your setup:

Alerting with Alertmanager

Alertmanager handles alerts sent by Prometheus:

1. Download and install Alertmanager:

   cd /opt
   sudo wget https://github.com/prometheus/alertmanager/releases/download/v0.28.1/alertmanager-0.28.1.linux-amd64.tar.gz
   sudo tar -xvzf alertmanager-0.28.1.linux-amd64.tar.gz

2. Configure Alertmanager in /etc/alertmanager/alertmanager.yml
3. Update your Prometheus configuration to include:

   alerting:
     alertmanagers:
       - static_configs:
           - targets:
             - localhost:9093

Remote Storage Integration

Prometheus can write data to remote storage systems for long-term retention:

remote_write:
  - url: "https://remote-storage-endpoint/api/v1/write"

remote_read:
  - url: "https://remote-storage-endpoint/api/v1/read"

Federation Setup

For monitoring multiple Prometheus servers:

scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="prometheus"}'
        - '{job="node", instance="server1:9100"}'
    static_configs:
      - targets:
        - 'prometheus-secondary:9090'

Service Discovery

Instead of static configurations, use dynamic service discovery:

scrape_configs:
  - job_name: 'file_sd'
    file_sd_configs:
      - files:
        - '/etc/prometheus/targets/*.json'
        refresh_interval: 5m

These advanced configurations will help you build a more flexible, scalable monitoring infrastructure that can grow with your needs.

Troubleshooting Common Issues

Even with careful installation, you might encounter issues with your Prometheus setup. Here are solutions to common problems:

Prometheus Service Won’t Start

If the Prometheus service fails to start:

1. Check for syntax errors in your configuration:

   sudo -u prometheus promtool check config /etc/prometheus/prometheus.yml

2. Verify permissions on directories and files:

   sudo ls -la /etc/prometheus
   sudo ls -la /var/lib/prometheus

3. Check the systemd logs:

   sudo journalctl -u prometheus -n 50 --no-pager

Cannot Access Web Interface

If you can’t access the Prometheus web interface:

1. Verify the service is running:

   sudo systemctl status prometheus

2. Check firewall settings:

   sudo firewall-cmd --list-all

3. Test the connection locally:

   curl http://localhost:9090

Targets Show as “Down”

If your monitoring targets appear as “down” in the Prometheus interface:

1. Verify the exporter service is running:

   sudo systemctl status node_exporter

2. Check network connectivity:

   curl http://localhost:9100/metrics

3. Inspect the Prometheus logs:

   sudo journalctl -u prometheus | grep scrape

High Disk Usage

If Prometheus is consuming too much disk space:

1. Adjust the retention period in prometheus.yml:

   global:
     scrape_interval: 15s
     evaluation_interval: 15s
     storage.tsdb.retention.time: 15d

2. Restart Prometheus after configuration changes:

   sudo systemctl restart prometheus

Understanding these troubleshooting techniques will help you maintain a healthy Prometheus installation on your CentOS Stream 10 system.

Congratulations! You have successfully installed Prometheus. Thanks for using this tutorial for installing the Prometheus monitoring system tool on your CentOS Stream 10 system. For additional help or useful information, we recommend you check the official Prometheus website.