How To Install Apache Kafka on Ubuntu 20.04 LTS

In this tutorial, we will show you how to install Apache Kafka on Ubuntu 20.04 LTS. For those of you who didn’t know, Apache Kafka is a distributed streaming platform that has become an essential component in modern data architectures. It enables the real-time processing, storage, and analysis of large volumes of data streams, making it a powerful tool for building scalable and fault-tolerant applications. Whether you’re working with IoT devices, log aggregation, or real-time analytics, Kafka provides a robust solution for handling high-throughput data streams.

This article assumes you have at least basic knowledge of Linux, know how to use the shell, and most importantly, you host your site on your own VPS. The installation is quite simple and assumes you are running in the root account, if not you may need to add ‘sudo‘ to the commands to get root privileges. I will show you through the step-by-step installation of Apache Kafka on Ubuntu 20.04 (Focal Fossa). You can follow the same instructions for Ubuntu 18.04, 16.04, and any other Debian-based distribution like Linux Mint.

Prerequisites

• A server running one of the following operating systems: Ubuntu 20.04, 18.04, 16.04, and any other Debian-based distribution like Linux Mint.
• It’s recommended that you use a fresh OS install to prevent any potential issues.
• A non-root sudo user or access to the root user. We recommend acting as a non-root sudo user, however, as you can harm your system if you’re not careful when acting as the root.

Install Apache Kafka on Ubuntu 20.04 LTS Focal Fossa

Step 1. First, make sure that all your system packages are up-to-date by running the following apt commands in the terminal.

sudo apt update
sudo apt upgrade

Step 2. Installing Java.

Apache Kafka requires Java to be installed on your server. Run the following command to install OpenJDK on the Ubuntu system:

sudo apt install openjdk-11-jdk

 Once the installation is complete, verify that Java is installed correctly by running:

$ java -version

java version "1.8.0_251"
Java(TM) SE Runtime Environment (build 1.8.0_251-b07)
Java HotSpot(TM) 64-Bit Server VM (build 25.251-b07, mixed mode)

Step 3. Installing Apache Kafka on Ubuntu 20.04.

Now we download the latest version of Kafka on the official web page:

wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz

Then, un-tar the archive file and move to another location:

sudo tar xzf kafka_2.13-3.7.0.tgz
sudo mv kafka_2.13-3.7.0 /opt/kafka

Step 4. Create Zookeeper and Kafka Systemd Unit Files.

Create the systemd unit file for zookeeper service:

sudo nano /etc/systemd/system/zookeeper.service

Add the following lines:

[Unit]
Description=Apache Zookeeper service
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Next, create the systemd unit file for Kafka service:

sudo nano /etc/systemd/system/kafka.service

Add the following lines:

[Unit]
Description=Apache Kafka Service
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/opt/jdk/jdk1.8.0_251"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

Reload the daemon to take effect:

sudo systemctl daemon-reload
sudo systemctl enable --now zookeeper
sudo systemctl enable --now kafka

Step 5. Creating Topic in Kafka.

Now we will create a topic named “idroot” with a single replication-factor and partition:

cd /opt/kafka
sudo bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-f

To send some messages for the created Topic:

sudo bin/kafka-console-producer.sh --broker-list localhost:9092 --topic idoot

It’s prompt for messages to type:

> Hi
> How are you?

Using the below command we can see the list of messages:

sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic idroot --from-beginning

Step 6. To Connect Kafka from remote machines.

To connect, create a Topic, and send messages from a remote server. Please follow the below steps:

cd /opt/kafka/config

Now look for server.properties and make some configuration changes:

sudo nano server.properties

listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://<HOST IP>:9092

Congratulations! You have successfully installed Apache Kafka. Thanks for using this tutorial for installing Apache Kafka on Ubuntu 20.04 LTS Focal Fossa system. For additional help or useful information, we recommend you check the official Apache Kafka website.