In this tutorial, we will show you how to install Apache Kafka on Rocky Linux 9. For those of you who didn’t know, Apache Kafka is open-source software that enables the storage and processing of data streams via a distributed streaming platform. It’s developed by Apache Software Foundation and written in Java and Scala. Apache Kafka is used to build real-time streaming data pipelines and applications that adapt to the data stream, especially for enterprise-grade applications and mission-critical applications.
This article assumes you have at least basic knowledge of Linux, know how to use the shell, and most importantly, you host your site on your own VPS. The installation is quite simple and assumes you are running in the root account, if not you may need to add ‘
sudo‘ to the commands to get root privileges. I will show you the step-by-step installation of Apache Kafka distributed streaming platform. on Rocky Linux. 9.
- A server running one of the following operating systems: Rocky Linux 9.
- It’s recommended that you use a fresh OS install to prevent any potential issues.
- SSH access to the server (or just open Terminal if you’re on a desktop).
non-root sudo useror access to the
root user. We recommend acting as a
non-root sudo user, however, as you can harm your system if you’re not careful when acting as the root.
Install Apache Kafka on Rocky Linux 9
Step 1. The first step is to update your system to the latest version of the package list. To do so, run the following commands:
sudo dnf check-update sudo dnf install dnf-utils sudo dnf install epel-release
Step 2. Installing Java.
Apache Kafka is a Java-based application, so Java must be installed on your server. Now run the following command below to install Java to your system:
sudo dnf install java-11-openjdk
Verify the Java installation using the following command:
For additional resources on installing Java, read the post below:
Step 3. Installing Apache Kafka on Rocky Linux 9.
First, we create a new system user for Kafka with the default home directory ‘
sudo useradd -r -d /opt/kafka -s /usr/sbin/nologin kafka
By default, Apache Kafka is not available on Rocky Linux 9 base repository. Now run the following command below to download the latest version of Apache Kafka to your system:
cd /opt sudo curl -fsSLo kafka.tgz https://downloads.apache.org/kafka/3.3.1/kafka_2.12-3.3.1.tgz
Next, Extract the file downloaded:
tar -xzf kafka.tgz sudo mv kafka_2.12-3.3.1 /opt/kafka
We will need to change some folders permissions:
sudo chown -R kafka:kafka /opt/kafka
Then, we create a new logs directory for Apache Kafka. Also, edit the default configuration ‘
server.properties‘ using favorite text editor:
sudo -u kafka mkdir -p /opt/kafka/logs sudo -u kafka nano /opt/kafka/config/server.properties
Uncomment the ‘
log.dirs‘ option and change the value to ‘
# logs configuration for Apache Kafka log.dirs=/opt/kafka/logs
Step 4. Create Systemd for Apache Kafka and Zookeeper.
For the production environment, it is recommended to create a systemd service file to run both Zookeeper and Kafka in the background.
Now we create a
systemd service file for Zookeeper with the following command:
sudo nano /etc/systemd/system/zookeeper.service
Add the following file:
[Unit] Requires=network.target remote-fs.target After=network.target remote-fs.target [Service] Type=simple User=kafka ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target
After that, Next, create a new service file for Apache Kafka:
sudo nano /etc/systemd/system/kafka.service
Add the following file:
[Unit] Requires=zookeeper.service After=zookeeper.service [Service] Type=simple User=kafka ExecStart=/bin/sh -c '/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties > /opt/kafka/logs/start-kafka.log 2>&1' ExecStop=/opt/kafka/bin/kafka-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target
Save changes and exit the file, then start and enable the Apache Kafka service using the commands below:
sudo systemctl daemon-reload sudo systemctl start zookeeper sudo systemctl start kafka
Next, enable both kafka and zookeeper services to be run automatically at system boot via the systemctl command below.
sudo systemctl enable zookeeper sudo systemctl enable kafka
Step 5. Create Test Topics on Kafka.
Kafka allows us to read, write, store, and process events across the various machines, however, to store these events we need someplace or folder and that called “
Topics“. Now we change the directory to Apache Kafka and create a test topic named topic1 with the following command:
cd /usr/local/kafka/ bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic topic1
Now verify your created topic using the following command:
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
First, run the following command to create an event named event1 using the following command below:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic event1
Next, type some text that you want to stream and display on the Consumer:
>Hi, this is my first event
Open another terminal and run the following command to display the generated event data in real-time:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic event1 --from-beginning
You will get the following output:
Hi, this is my first event
Congratulations! You have successfully installed Apache Kafka. Thanks for using this tutorial for installing Apache Kafka distributed streaming platform. on your Rocky Linux 9 system. For additional help or useful information, we recommend you check the official Apache website.