How To Install Prometheus on AlmaLinux 10

Prometheus has become the industry standard for monitoring modern infrastructure and cloud-native applications. This open-source monitoring system excels at collecting and storing metrics as time-series data, making it invaluable for tracking system performance, application behavior, and infrastructure health. AlmaLinux 10, as a robust enterprise-grade Linux distribution and CentOS successor, provides an excellent platform for hosting Prometheus monitoring infrastructure.

This comprehensive guide walks through every step of installing and configuring Prometheus on AlmaLinux 10. From initial system preparation to securing and optimizing the installation, each section provides detailed instructions with explanations of why each step matters. Whether managing a single server or planning enterprise-wide infrastructure monitoring, this tutorial delivers the knowledge needed for a production-ready Prometheus deployment.

Prerequisites and System Requirements

Before beginning the Prometheus installation process, ensure the system meets all necessary requirements.

Server Requirements

A fresh AlmaLinux 10 installation serves as the ideal starting point. For testing environments, allocate at least 2 CPU cores, 4GB RAM, and 20GB disk space. Production deployments demand more substantial resources—4 or more CPU cores with 8GB RAM minimum. Storage requirements scale with retention needs and the number of monitored targets. Plan for approximately 1-2 bytes per sample, multiplied by samples per second and retention period.

Access and Privileges

Root access or a user account with sudo privileges is essential. SSH access to the server enables remote management. Basic familiarity with Linux command-line operations, systemd service management, and fundamental networking concepts ensures smooth progress through installation steps.

Network Requirements

Configure a static IP address for production systems to maintain consistent access. The firewall requires configuration to allow Prometheus ports. Active internet connectivity enables downloading necessary packages and binaries. For security-conscious environments, consider implementing VPN access or reverse proxy configurations.

Understanding Prometheus Architecture

Prometheus follows a pull-based monitoring model, fundamentally different from traditional push-based systems.

Core Components

The Prometheus server forms the heart of the monitoring stack, scraping metrics from configured targets at regular intervals. Its built-in time-series database (TSDB) stores metrics efficiently using a custom storage engine optimized for time-series data. PromQL, the powerful query language, enables flexible metric analysis and aggregation. The integrated web UI provides immediate access to queries, graphs, and system status.

How Prometheus Works

Targets expose metrics via HTTP endpoints. Prometheus scrapes these endpoints at configured intervals, typically every 15 or 30 seconds. The scraped metrics are stored in the local TSDB with timestamps and labels. This pull model offers several advantages: centralized configuration, easier debugging of metric collection, and reduced load on monitored systems.

Key Features

Multi-dimensional data modeling allows precise metric identification through labels. Unlike traditional monitoring systems that rely on hierarchical naming schemes, Prometheus uses key-value pairs for maximum flexibility. The system operates independently without requiring distributed storage, simplifying deployment and maintenance. Service discovery mechanisms automatically identify monitoring targets in dynamic environments.

Step 1: Preparing the AlmaLinux 10 Environment

System preparation ensures a clean foundation for Prometheus installation.

Update System Packages

Begin by updating all installed packages to their latest versions:

sudo dnf update -y

This command updates package repositories and installs available updates. The process typically completes within a few minutes depending on internet speed and the number of packages requiring updates. Keeping the system current addresses security vulnerabilities and ensures compatibility with new software.

Install Required Dependencies

Install essential tools for downloading and managing Prometheus:

sudo dnf install wget curl nano tar -y

Each tool serves a specific purpose. Wget downloads files from the internet. Curl performs API requests and file transfers. Vim enables configuration file editing. Tar extracts compressed archives containing Prometheus binaries.

Configure SELinux

Check SELinux status:

sestatus

For testing environments, temporarily set SELinux to permissive mode if encountering permission issues. Production systems should maintain enforcing mode with properly configured policies. Create custom SELinux policies for Prometheus if required by organizational security standards.

Verify System Resources

Confirm adequate resources before proceeding:

df -h
free -h
nproc

These commands display available disk space, memory, and CPU cores respectively. Insufficient resources lead to performance degradation or installation failures.

Step 2: Creating Prometheus System User and Directory Structure

Security best practices mandate running services with dedicated system accounts rather than root privileges.

Create Prometheus System User

Execute the following command to create a system user for Prometheus:

sudo useradd --no-create-home --shell /bin/false prometheus

The --no-create-home flag prevents creating a home directory since Prometheus doesn’t require user login capabilities. The --shell /bin/false parameter disables interactive shell access, preventing potential security risks. This approach implements the principle of least privilege, granting only permissions necessary for service operation.

Verify user creation:

id prometheus

The output should display the user ID and group ID for the prometheus account.

Create Directory Structure

Establish directories for configuration files and data storage:

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus

The /etc/prometheus directory stores configuration files, rules, and console templates. The /var/lib/prometheus directory holds time-series data collected from monitored targets.

Set Directory Ownership

Assign appropriate ownership to ensure Prometheus can access these directories:

sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Proper ownership prevents permission-denied errors during service startup. The Prometheus process runs as the prometheus user and must write to these locations.

Configure Directory Permissions

Set secure permissions:

sudo chmod 755 /etc/prometheus
sudo chmod 755 /var/lib/prometheus

These permissions allow the prometheus user full access while restricting modifications by other users.

Step 3: Downloading and Installing Prometheus

Obtain the latest Prometheus release from the official GitHub repository.

Identify Current Version

Visit the Prometheus GitHub releases page or use curl to check:

curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest | grep tag_name

This command queries the GitHub API for the most recent stable release. At the time of writing, Prometheus 2.45.0 represents a stable production-ready version.

Download Prometheus Binary

Navigate to a temporary directory and download Prometheus:

cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v3.6.0/prometheus-3.6.0.linux-amd64.tar.gz

Replace the version number with the latest available release. The download size typically ranges from 80-100MB.

Extract the Archive

Extract the downloaded tarball:

tar -xvf prometheus-3.6.0.linux-amd64.tar.gz
cd prometheus-3.6.0.linux-amd64

The extraction creates a directory containing Prometheus binaries, configuration files, and console templates.

Install Prometheus Binaries

Copy the prometheus and promtool binaries to the system path:

sudo cp prometheus /usr/local/bin/
sudo cp promtool /usr/local/bin/

Placing binaries in /usr/local/bin makes them accessible system-wide without specifying full paths.

Install Console Files

Copy console templates and libraries:

sudo cp -r consoles /etc/prometheus
sudo cp -r console_libraries /etc/prometheus

Console templates provide pre-built web interfaces for viewing common metrics.

Set Binary Permissions

Ensure proper ownership and execution permissions:

sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo chmod +x /usr/local/bin/prometheus
sudo chmod +x /usr/local/bin/promtool

Verify Installation

Confirm successful installation:

prometheus --version
promtool --version

Both commands should display version information, confirming proper installation.

Clean Up

Remove downloaded files to free disk space:

cd /tmp
rm -rf prometheus-3.6.0.linux-amd64*

Step 4: Configuring Prometheus

Configuration determines which targets Prometheus monitors and how it operates.

Create Configuration File

Create the main configuration file at /etc/prometheus/prometheus.yml:

sudo nano /etc/prometheus/prometheus.yml

Basic Configuration Structure

Add the following configuration:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    monitor: 'prometheus-monitor'

alerting:
  alertmanagers:
    - static_configs:
        - targets: []

rule_files:
  # - "alerts.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

Configuration Explanation

The global section defines default parameters for all scrape operations. The scrape_interval determines how frequently Prometheus collects metrics from targets. Setting this to 15 seconds balances resource usage with data granularity. The evaluation_interval controls how often Prometheus evaluates alerting rules.

External labels attach metadata to all time series and alerts sent to external systems. These prove valuable when federating multiple Prometheus instances or sending data to remote storage.

The alerting section configures Alertmanager integration. Leave this empty initially and configure it later when implementing alerting capabilities.

The rule_files section specifies files containing recording and alerting rules. Uncomment and add rule files as monitoring requirements evolve.

Scrape configurations define monitored targets. The first job monitors Prometheus itself, providing insight into the monitoring system’s health. The second job prepares for Node Exporter installation in subsequent steps.

Validate Configuration

Check configuration syntax:

promtool check config /etc/prometheus/prometheus.yml

This command validates YAML syntax and configuration logic, reporting any errors or warnings.

Set Configuration Permissions

Ensure proper ownership:

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
sudo chmod 644 /etc/prometheus/prometheus.yml

Step 5: Creating Prometheus Systemd Service

Systemd integration enables automatic startup, process management, and centralized logging.

Create Service Unit File

Create the systemd service file:

sudo nano /etc/systemd/system/prometheus.service

Service Configuration

Add the following content:

[Unit]
Description=Prometheus Monitoring System
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --storage.tsdb.retention.time=15d

SyslogIdentifier=prometheus
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Service File Explanation

The [Unit] section defines service metadata and dependencies. Wants=network-online.target ensures network availability before starting Prometheus. The After directive specifies startup order.

The [Service] section controls process execution. Running as the prometheus user enhances security. The ExecStart line launches Prometheus with essential flags:

• --config.file specifies the configuration file location
• --storage.tsdb.path sets the data directory
• --web.console.templates and --web.console.libraries point to console resources
• --web.listen-address binds Prometheus to all network interfaces on port 9090
• --storage.tsdb.retention.time sets data retention to 15 days

The Restart=always directive ensures automatic recovery from crashes. RestartSec=5 introduces a 5-second delay between restart attempts, preventing rapid restart loops.

The [Install] section determines when systemd enables the service during boot.

Step 6: Starting and Enabling Prometheus Service

With configuration complete, activate the Prometheus service.

Reload Systemd Daemon

Inform systemd about the new service file:

sudo systemctl daemon-reload

This command parses all service files, registering the new Prometheus service.

Start Prometheus

Initiate the Prometheus service:

sudo systemctl start prometheus

Check Service Status

Verify successful startup:

sudo systemctl status prometheus

A successful start displays “active (running)” in green text. The output includes the process ID, memory usage, and recent log entries.

Enable Automatic Startup

Configure Prometheus to start automatically on system boot:

sudo systemctl enable prometheus

This creates symbolic links in systemd directories, ensuring Prometheus starts after reboots.

View Service Logs

Monitor Prometheus logs in real-time:

sudo journalctl -u prometheus -f

Press Ctrl+C to exit the log viewer. Review logs for any warning or error messages requiring attention.

Troubleshoot Startup Issues

If the service fails to start, examine logs for error messages. Common issues include:

• Configuration syntax errors detected by promtool
• Permission problems accessing directories or configuration files
• Port 9090 already in use by another service
• Missing or incorrect file paths in the service file

Resolve identified issues and restart the service.

Step 7: Configuring Firewall Rules

Enable external access to Prometheus through firewall configuration.

Check Firewall Status

Verify firewalld is active:

sudo firewall-cmd --state

Add Prometheus Port

Open port 9090 for Prometheus web interface access:

sudo firewall-cmd --zone=public --add-port=9090/tcp --permanent

The --permanent flag persists the rule across firewall restarts and system reboots.

Reload Firewall

Apply the new rule:

sudo firewall-cmd --reload

Verify Firewall Configuration

List active firewall rules:

sudo firewall-cmd --list-all

Confirm port 9090/tcp appears in the ports list.

Security Considerations

For production environments, restrict access to specific IP addresses or subnets using rich rules:

sudo firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" port protocol="tcp" port="9090" accept' --permanent

This example permits access only from the 192.168.1.0/24 subnet. Adjust the IP range to match organizational requirements.

Step 8: Accessing Prometheus Web Interface

The Prometheus web interface provides query execution, metric exploration, and system status monitoring.

Access the Web UI

Open a web browser and navigate to:

http://SERVER_IP:9090

Replace SERVER_IP with the actual IP address of the AlmaLinux server. The Prometheus homepage displays the query interface.

Explore the Interface

The navigation menu includes several important sections:

• Graph: Execute PromQL queries and visualize results
• Alerts: View active, pending, and silenced alerts
• Status: Access configuration, targets, and service discovery information
• Help: Quick reference for PromQL syntax

Verify Target Status

Navigate to Status → Targets. The prometheus job should display with an “UP” state. This confirms Prometheus successfully scrapes its own metrics.

Execute Test Queries

Enter the following queries in the expression browser:

up

This query returns 1 for all targets currently reachable. It provides a quick health check for monitored systems.

prometheus_http_requests_total

This counter tracks total HTTP requests received by Prometheus, demonstrating metric collection is functioning.

Understanding Metrics

Prometheus supports four metric types:

• Counter: Monotonically increasing values (requests, errors)
• Gauge: Values that increase or decrease (temperature, memory usage)
• Histogram: Observations bucketed by configurable ranges (request duration)
• Summary: Similar to histogram with client-side percentile calculation

Step 9: Installing and Configuring Node Exporter

Node Exporter collects hardware and operating system metrics from Linux systems.

Download Node Exporter

Navigate to the temporary directory:

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.9.1/node_exporter-1.9.1.linux-amd64.tar.gz

Extract and Install

Extract the archive:

tar -xvf node_exporter-1.9.1.linux-amd64.tar.gz

Copy the binary:

sudo cp node_exporter-1.9.1.linux-amd64/node_exporter /usr/local/bin/

Set ownership and permissions:

sudo chown prometheus:prometheus /usr/local/bin/node_exporter
sudo chmod +x /usr/local/bin/node_exporter

Create Node Exporter Service

Create the systemd service file:

sudo nano /etc/systemd/system/node_exporter.service

Add the following configuration:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/node_exporter
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Start Node Exporter

Reload systemd and start the service:

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
sudo systemctl status node_exporter

Configure Firewall for Node Exporter

Open port 9100:

sudo firewall-cmd --zone=public --add-port=9100/tcp --permanent
sudo firewall-cmd --reload

Reload Prometheus Configuration

Restart Prometheus to pick up the Node Exporter target:

sudo systemctl restart prometheus

Verify Node Exporter Metrics

Access the Prometheus web interface and navigate to Status → Targets. The node_exporter job should appear with “UP” status.

Execute sample queries:

node_memory_MemAvailable_bytes
node_cpu_seconds_total
node_filesystem_avail_bytes

These queries display available memory, CPU time, and filesystem space respectively.

Securing Prometheus Installation

Production deployments require additional security measures beyond basic installation.

Implement Reverse Proxy Authentication

Configure Nginx or Apache as a reverse proxy with basic authentication:

sudo dnf install nginx -y

Create a password file:

sudo htpasswd -c /etc/nginx/.htpasswd admin

Configure Nginx to proxy requests to Prometheus with authentication enabled.

Configure SSL/TLS

Generate SSL certificates using Let’s Encrypt or organizational certificate authority. Configure the reverse proxy to handle HTTPS connections, encrypting data in transit.

Network Security Hardening

Implement IP whitelisting at the firewall level. Consider VPN-only access for production monitoring systems. Regularly audit firewall rules to ensure compliance with security policies.

File Permission Audit

Regularly review file permissions on configuration files and binaries. Restrict read access to configuration files containing sensitive information.

Optimizing Prometheus Performance

Proper tuning ensures efficient operation at scale.

Configure Data Retention

Adjust retention time based on storage capacity and requirements:

--storage.tsdb.retention.time=30d
--storage.tsdb.retention.size=50GB

Shorter retention periods reduce storage requirements. Longer retention enables historical analysis.

Memory Optimization

Prometheus memory usage correlates with active time series. Monitor memory consumption:

process_resident_memory_bytes

For systems with limited memory, reduce scrape frequency or the number of monitored targets.

Query Performance

Write efficient PromQL queries avoiding unbounded time ranges. Use recording rules for frequently executed complex queries. Recording rules pre-compute expensive queries at regular intervals, storing results as new time series.

Scrape Interval Optimization

Balance data granularity against resource consumption. High-frequency scraping (5-10 seconds) provides detailed metrics but increases CPU and memory usage. Lower-frequency scraping (30-60 seconds) reduces load at the cost of metric resolution.

Maintenance and Best Practices

Regular maintenance ensures continued reliable operation.

Backup Procedures

Backup the following components regularly:

• Configuration files in /etc/prometheus
• Time-series data in /var/lib/prometheus
• Custom rules and alert definitions

Implement automated backup scripts running via cron. Test restoration procedures periodically.

Update Procedures

Monitor Prometheus releases for security updates and new features. Test updates in non-production environments before deploying to production. Maintain rollback capability by preserving previous binary versions.

Configuration Management

Store configuration files in version control systems like Git. Document all changes with descriptive commit messages. Implement peer review for configuration modifications.

Monitoring Best Practices

Apply consistent naming conventions to metrics and labels. Avoid high-cardinality labels that create excessive unique time series. Document custom metrics and their meanings for team members.

Troubleshooting Common Issues

Address frequent problems encountered during operation.

Service Startup Failures

Check systemd status and logs for error messages. Common causes include configuration syntax errors, permission issues, and port conflicts. Resolve identified problems and restart the service.

High Resource Usage

Excessive memory consumption often results from too many active time series. Query the time series count:

count({__name__=~".+"})

Reduce cardinality by eliminating unnecessary labels or decreasing retention time.

Target Scraping Failures

When targets show “DOWN” status, verify network connectivity between Prometheus and the target. Check firewall rules on both systems. Confirm the target service is running and exposing metrics on the expected port.

Storage Issues

Monitor available disk space in the data directory. Prometheus automatically removes old data based on retention settings. If storage fills prematurely, reduce retention time or increase available disk space.

Congratulations! You have successfully installed Prometheus. Thanks for using this tutorial for installing the Prometheus monitoring system tool on your AlmaLinux OS 10 system. For additional help or useful information, we recommend you check the official Prometheus website.