How To View Systemctl Logs

Understanding system logs is crucial for effective Linux system administration. Systemctl logs provide invaluable insights into your system’s health, performance, and potential issues. Whether you’re troubleshooting a failed service, monitoring system performance, or conducting routine maintenance, knowing how to view and interpret systemctl logs is an essential skill for any Linux administrator.

In this comprehensive guide, we’ll explore various methods to view, filter, and analyze systemctl logs. From basic commands to advanced filtering techniques, you’ll learn everything you need to become proficient at log analysis in Linux systems using systemd.

Understanding Systemd and Systemctl Logs

Systemd is the modern initialization system and service manager for Linux operating systems. It has replaced older systems like SysVinit and Upstart in most mainstream Linux distributions. Systemd is responsible for bootstrapping the user space and managing system processes after the Linux kernel has started.

What Makes Systemd Different?

Unlike traditional text-based logging systems that write to plain text files in the /var/log directory, systemd uses a centralized logging system called systemd-journald. This service collects logs from various sources, including:

• Kernel messages
• System service logs
• Application logs
• Boot messages
• Standard output and standard error of services

The systemd journal stores logs in a structured, binary format rather than plain text. This format provides several advantages, including indexed searches, structured metadata, and improved performance for log analysis.

Log Storage Locations

Systemd logs are typically stored in one of two locations:

• /var/log/journal/ for persistent storage (survives system reboots)
• /run/log/journal/ for volatile storage (cleared on reboot)

By default, many systems use volatile storage to save disk space. To enable persistent logs, you need to create the journal directory and restart the systemd-journald service.

Log Structure and Components

Each log entry in systemd contains several fields of information, making them easy to understand and filter:

• Timestamp: When the event occurred
• Hostname: The system that generated the log
• Service or process name: What generated the log
• Process ID (PID): The specific process ID
• Log message: The actual content of the log
• Priority level: Indicating severity, from emergency (0) to debug (7)

This structured approach makes systemd logs more powerful for troubleshooting and monitoring compared to traditional text-based logs.

Basic Commands for Viewing Systemctl Logs

The primary tool for accessing systemd logs is journalctl. This command-line utility provides a comprehensive interface to the systemd journal. Let’s explore the basic usage patterns.

Viewing All Logs

To display all journal entries in your system, simply run:

journalctl

This command outputs all available logs in chronological order, starting with the oldest entries. The output is paginated, similar to the less command, allowing you to navigate up and down using arrow keys.

Understanding the Output Format

The default output of journalctl displays several columns of information:

Apr 05 06:30:15 hostname systemd[1]: Started Daily apt download activities.

This example shows:

• Date and time (Apr 05 06:30:15)
• Hostname
• Service/process name (systemd)
• PID in square brackets
• The actual log message

Checking Service Status

For a quick overview of a service’s status and its recent logs, use:

systemctl status service-name

This provides essential information including whether the service is active, enabled, its PID, recent log entries, and more. While useful for a quick check, it only shows a limited number of recent log entries.

Navigating the Log Output

When using journalctl, you can:

• Press Space or Page Down to move forward
• Press b or Page Up to move backward
• Press q to exit the log viewer
• Press / to search for text
• Press n to move to the next search result

These navigation options make it easier to browse through large log files effectively.

Viewing Recent and Real-time Logs

Often, you’ll be most interested in the most recent logs or want to monitor logs in real-time as they’re generated.

Displaying Most Recent Entries

To view only the most recent log entries, use the -n option followed by the number of entries you want to see:

journalctl -n 20

This shows the last 20 log entries. If you don’t specify a number, it defaults to 10 entries.

Jumping to the End of Logs

To immediately jump to the end of the log file and see the most recent entries, use:

journalctl -e

This is particularly useful when dealing with large log files where scrolling would be impractical.

Following Logs in Real-time

To monitor logs as they’re generated in real-time (similar to tail -f), use:

journalctl -f

This command continuously displays new log entries as they’re added to the journal. It’s invaluable for troubleshooting active issues or monitoring service behavior during testing. Press Ctrl+C to exit the real-time view.

Combining Options for Effective Monitoring

You can combine these options for more specific monitoring. For example, to follow only the most recent logs for a specific service:

journalctl -f -u nginx -n 50

This displays the last 50 log entries for the nginx service and continues to show new entries as they appear.

Filtering Logs by Service Unit

One of the most useful ways to filter logs is by service unit, allowing you to focus on specific services rather than wading through all system logs.

Viewing Service-specific Logs

To view logs for a particular service, use the -u option followed by the service name:

journalctl -u nginx

This displays all journal entries related to the specified service, making troubleshooting much more manageable.

Multiple Service Filtering

You can monitor multiple services simultaneously by specifying multiple -u options:

journalctl -u nginx -u apache2 -u mysql

This command shows logs from all three services, helping you identify potential interactions between them.

Using Wildcards for Service Groups

To view logs for services with similar names, you can use wildcards:

journalctl -u systemd-*

This shows logs for all services that start with “systemd-“, such as systemd-networkd, systemd-resolved, etc.

User Service Logs

For user-specific services rather than system services, use:

journalctl --user-unit my-application

This shows logs for user services, which run in the user’s context rather than as system services. Note that you can only see logs for units associated with your own user account with this command.

Finding Available Service Units

To discover what service units are available on your system, use:

systemctl list-units --type=service

This lists all active service units, providing names you can use with the -u option in journalctl.

Time-based Log Filtering

Time-based filtering allows you to focus on logs from specific time periods, which is essential for troubleshooting incidents that occurred at known times.

Boot-specific Logs

To view logs from the current boot session only:

journalctl -b

For logs from previous boot sessions, use -b with a negative offset:

journalctl -b -1    # Previous boot
journalctl -b -2    # Second-to-last boot

This is extremely useful when troubleshooting boot-related issues or problems that started after a system restart.

Date and Time Filtering

To filter logs within a specific time range, use the --since and --until options:

journalctl --since "2025-04-04 12:00:00" --until "2025-04-05 06:00:00"

You can also use relative time expressions:

journalctl --since "yesterday"
journalctl --since "1 hour ago"

These options allow you to narrow down logs to specific incidents or time periods.

Combining Time Filters with Service Filters

For even more precise analysis, combine time-based filters with service filters:

journalctl -u nginx --since "today" --until "1 hour ago"

This shows nginx logs from today up until one hour ago, providing a focused view for troubleshooting recent issues.

Priority-based Log Filtering

Systemd logs have priority levels ranging from emergency (0) to debug (7). Filtering by priority lets you focus on critical issues without being distracted by less important messages.

Understanding Priority Levels

The priority levels in systemd are:

• 0: emerg (Emergency) – System is unusable
• 1: alert (Alert) – Action must be taken immediately
• 2: crit (Critical) – Critical conditions
• 3: err (Error) – Error conditions
• 4: warning (Warning) – Warning conditions
• 5: notice (Notice) – Normal but significant conditions
• 6: info (Informational) – Informational messages
• 7: debug (Debug) – Debug-level messages

Filtering by Priority Level

To filter logs by priority, use the -p option:

journalctl -p err

This shows only logs with priority level “error” (3) and higher (0-3). You can specify the priority level either by name or number.

Setting Priority Ranges

You can also specify a range of priorities:

journalctl -p err..warning

This shows logs with priorities from error (3) to warning (4).

Combining with Other Filters

For advanced troubleshooting, combine priority filtering with other filters:

journalctl -p err -u nginx --since "today"

This shows only error-level or higher logs from the nginx service that occurred today, helping you quickly identify critical issues.

Content-based Log Filtering

When looking for specific issues, you might need to filter logs based on their content. This can be done using text searches and pattern matching.

Basic Keyword Searching

While viewing logs with journalctl, you can search for keywords by pressing / followed by your search term. However, for more powerful filtering, you can use grep:

journalctl | grep "connection refused"

This filters the logs to show only entries containing the exact phrase “connection refused”.

Regular Expression Matching

For more complex pattern matching, use regular expressions with grep:

journalctl | grep -E "(error|warning): .*database"

This finds all log entries containing either “error:” or “warning:” followed by the word “database” anywhere in the line.

Common Search Patterns

Some useful search patterns include:

• failed or failure for service start failures
• error or exception for general errors
• denied or permission for permission issues
• segfault for application crashes
• timeout for connection or operation timeouts

Combining Content Filters with Other Options

For more precise searches, combine content filtering with service and time filters:

journalctl -u nginx --since "today" | grep "404"

This shows nginx logs from today that include “404” (page not found errors), helping you identify problematic URLs.

Advanced Filtering Techniques

Beyond the basic filtering methods, journalctl supports advanced filtering techniques that can help you pinpoint specific issues more effectively.

Filtering by User ID

To view logs generated by a specific user:

journalctl _UID=1000

This shows all logs associated with the user whose UID is 1000.

Process ID Filtering

To view logs from a specific process:

journalctl _PID=1234

This displays logs generated by the process with PID 1234, useful when troubleshooting a specific application instance.

Kernel Message Filtering

To view only kernel messages:

journalctl -k

This is equivalent to the traditional dmesg command but with the additional features of journalctl.

Hardware-related Logs

To focus on hardware-related issues:

journalctl --dmesg | grep -i "usb\|disk\|cpu\|memory"

This filters kernel messages for common hardware-related terms.

Combining Multiple Filters

You can combine multiple filtering criteria for highly targeted log analysis:

journalctl _UID=33 -u apache2 -p err --since "2 days ago"

This command shows error-level logs from the apache2 service that were generated by the user with UID 33 (typically the www-data user on Debian-based systems) in the last two days.

Output Formatting Options

Journalctl supports various output formats to make logs easier to read or process programmatically.

Available Output Formats

To specify a particular output format, use the -o option followed by the format name:

journalctl -o json

Common formats include:

• short: The default format, similar to traditional syslog
• verbose: Shows all fields for each entry
• json: Outputs in JSON format for programmatic processing
• json-pretty: Formatted JSON output for better readability
• cat: Shows only the message field without timestamp or metadata

JSON Output for Programmatic Analysis

For scripting or automated log analysis, the JSON format is particularly useful:

journalctl -u nginx -o json > nginx_logs.json

This saves nginx logs in JSON format to a file, which can then be parsed by scripts or imported into log analysis tools.

Verbose Output for Detailed Analysis

When you need to see all metadata associated with log entries:

journalctl -o verbose

This shows all fields stored for each log entry, including those not visible in the default output format.

Log Management and Maintenance

Proper log management is essential to prevent logs from consuming excessive disk space while still retaining important information for troubleshooting.

Checking Journal Disk Usage

To see how much disk space is being used by the journal:

journalctl --disk-usage

This helps you monitor log growth and determine if cleanup is necessary.

Setting Journal Size Limits

You can configure the maximum size of the journal in /etc/systemd/journald.conf:

SystemMaxUse=1G

This limits the journal to a maximum of 1 gigabyte, automatically removing older entries when the limit is reached.

Clearing Old Logs

To manually clear old logs based on time:

journalctl --vacuum-time=2weeks

This removes journal entries older than two weeks. You can also clear based on size:

journalctl --vacuum-size=500M

This trims the journal to approximately 500 megabytes by removing older entries.

Enabling Persistent Journals

To ensure logs survive system reboots, create the journal directory:

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

This enables persistent storage of journal logs rather than keeping them only in volatile memory.

Troubleshooting with Systemctl Logs

Effective log analysis is crucial for troubleshooting various system issues. Here are some common scenarios and how to approach them using systemctl logs.

Diagnosing Service Failures

When a service fails to start, check its status and logs:

systemctl status service-name
journalctl -u service-name -e

Look for error messages indicating why the service failed, such as missing dependencies, configuration errors, or permission issues.

Identifying Startup Problems

For issues during system boot:

journalctl -b -p err

This shows error-level messages from the current boot, helping you identify services that failed during startup.

Debugging Configuration Errors

When a service starts but doesn’t work correctly:

journalctl -u service-name --since "10 minutes ago"

Look for warning or error messages that might indicate configuration problems. Often, the service will log specific details about what configuration options are incorrect or missing.

Correlating Events Across Services

For complex issues involving multiple services:

journalctl -u service1 -u service2 --since "1 hour ago"

This helps you see how events in one service might affect another, such as a database failure causing a web application to throw errors.

Interpreting Common Error Messages

Some frequently encountered error patterns and their likely causes:

• “Failed to start due to missing dependency” – Check the required dependency service
• “Permission denied” – File or directory permission issues
• “Configuration file error” – Syntax error in config files
• “Out of memory” – Service is using excessive memory
• “Connection refused” – Network connectivity or firewall issues

Best Practices for System Log Analysis

Adopting these best practices will help you maintain effective log monitoring and troubleshooting workflows.

Regular Monitoring Routines

Establish a regular schedule for checking critical service logs:

journalctl -u critical-service1 -u critical-service2 -p warning --since "yesterday"

Running this daily helps identify emerging issues before they become critical problems.

Using Log Forwarding

For centralized log management across multiple systems, forward systemd logs to a central log server using tools like rsyslog or a dedicated log management platform.

Security Considerations

Remember that logs may contain sensitive information. Protect access to journal logs:

sudo usermod -a -G systemd-journal username

This adds a user to the systemd-journal group, allowing them to read logs without full root access.

Documenting Troubleshooting Patterns

When you solve a problem using log analysis, document the specific log patterns and solutions for future reference. This builds a knowledge base for faster troubleshooting.