How To Install Apache Kafka on Rocky Linux 9

In this tutorial, we will show you how to install Apache Kafka on Rocky Linux 9. For those of you who didn’t know, Apache Kafka is open-source software that enables the storage and processing of data streams via a distributed streaming platform. It’s developed by Apache Software Foundation and written in Java and Scala. Apache Kafka is used to build real-time streaming data pipelines and applications that adapt to the data stream, especially for enterprise-grade applications and mission-critical applications.

This article assumes you have at least basic knowledge of Linux, know how to use the shell, and most importantly, you host your site on your own VPS. The installation is quite simple and assumes you are running in the root account, if not you may need to add ‘sudo‘ to the commands to get root privileges. I will show you the step-by-step installation of Apache Kafka distributed streaming platform. on Rocky Linux. 9.

Prerequisites

• A server running one of the following operating systems: Rocky Linux 9.
• It’s recommended that you use a fresh OS install to prevent any potential issues.
• SSH access to the server (or just open Terminal if you’re on a desktop).
• A non-root sudo user or access to the root user. We recommend acting as a non-root sudo user, however, as you can harm your system if you’re not careful when acting as the root.

Install Apache Kafka on Rocky Linux 9

Step 1. The first step is to update your system to the latest version of the package list. To do so, run the following commands:

sudo dnf check-update
sudo dnf install dnf-utils
sudo dnf install epel-release

Step 2. Installing Java.

Apache Kafka is a Java-based application, so Java must be installed on your server. Now run the following command below to install Java to your system:

sudo dnf install java-11-openjdk

Verify the Java installation using the following command:

java --version

For additional resources on installing Java, read the post below:

• How to install Java on Rocky Linux √

Step 3. Installing Apache Kafka on Rocky Linux 9.

First, we create a new system user for Kafka with the default home directory ‘/opt/kafka‘:

sudo useradd -r -d /opt/kafka -s /usr/sbin/nologin kafka

By default, Apache Kafka is not available on Rocky Linux 9 base repository. Now run the following command below to download the latest version of Apache Kafka to your system:

cd /opt
sudo curl -fsSLo kafka.tgz https://downloads.apache.org/kafka/3.3.1/kafka_2.12-3.3.1.tgz

Next, Extract the file downloaded:

tar -xzf kafka.tgz
sudo mv kafka_2.12-3.3.1 /opt/kafka

We will need to change some folders permissions:

sudo chown -R kafka:kafka /opt/kafka

Then, we create a new logs directory for Apache Kafka. Also, edit the default configuration ‘server.properties‘ using favorite text editor:

sudo -u kafka mkdir -p /opt/kafka/logs
sudo -u kafka nano /opt/kafka/config/server.properties

Uncomment the ‘log.dirs‘ option and change the value to ‘/opt/kafka/logs‘:

# logs configuration for Apache Kafka
log.dirs=/opt/kafka/logs

Step 4. Create Systemd for Apache Kafka and Zookeeper.

For the production environment, it is recommended to create a systemd service file to run both Zookeeper and Kafka in the background.

Now we create a systemd service file for Zookeeper with the following command:

sudo nano /etc/systemd/system/zookeeper.service

Add the following file:

[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=kafka
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

After that, Next, create a new service file for Apache Kafka:

sudo nano /etc/systemd/system/kafka.service

Add the following file:

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties > /opt/kafka/logs/start-kafka.log 2>&1'
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Save changes and exit the file, then start and enable the Apache Kafka service using the commands below:

sudo systemctl daemon-reload
sudo systemctl start zookeeper
sudo systemctl start kafka

Next, enable both kafka and zookeeper services to be run automatically at system boot via the systemctl command below.

sudo systemctl enable zookeeper
sudo systemctl enable kafka

Step 5. Create Test Topics on Kafka.

Kafka allows us to read, write, store, and process events across the various machines, however, to store these events we need someplace or folder and that called “Topics“. Now we change the directory to Apache Kafka and create a test topic named topic1 with the following command:

cd /usr/local/kafka/
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic topic1

Now verify your created topic using the following command:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Output:

topic1

First, run the following command to create an event named event1 using the following command below:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic event1

Next, type some text that you want to stream and display on the Consumer:

>Hi, this is my first event

Open another terminal and run the following command to display the generated event data in real-time:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic event1 --from-beginning

You will get the following output:

Hi, this is my first event

Congratulations! You have successfully installed Apache Kafka. Thanks for using this tutorial for installing Apache Kafka distributed streaming platform. on your Rocky Linux 9 system. For additional help or useful information, we recommend you check the official Apache website.