How To Install Apache Kafka on Debian 10

In this tutorial, we will show you how to install Apache Kafka on Debian 10. For those of you who didn’t know, Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

This article assumes you have at least basic knowledge of Linux, know how to use the shell, and most importantly, you host your site on your own VPS. The installation is quite simple and assumes you are running in the root account, if not you may need to add ‘sudo‘ to the commands to get root privileges. I will show you through the step-by-step installation of the Apache Kafka on a Debian 10 (Buster).

Prerequisites

• A server running one of the following operating systems: Debian 10 (Buster).
• It’s recommended that you use a fresh OS install to prevent any potential issues.
• A non-root sudo user or access to the root user. We recommend acting as a non-root sudo user, however, as you can harm your system if you’re not careful when acting as the root.

Install Apache Kafka on Debian 10 Buster

Step 1. Before running the tutorial below, it’s important to make sure your system is up to date by running the following apt commands in the terminal:

sudo apt update

Step 2. Installing Java.

Now install Java using the command below:

sudo apt install default-jdk

Verify the Java version by running the following command:

java -version

Step 3. Installing Apache Kafka on Debian 10.

Now we run the following command to download Apache Kafka packages from the official website:

wget https://downloads.apache.org/kafka/2.8.0/kafka_2.12-2.8.0.tgz

Then, extract the downloaded archive file:

tar xzf kafka_2.12-2.8.0.tgz
mv kafka_2.12-2.8.0 /usr/local/kafka

Step 4. Create Systemd Unit Files.

Now we create systemd unit files for the Zookeeper and Kafka service. This will help to manage Kafka services to start or stop using the systemctl command:

nano /etc/systemd/system/zookeeper.service

Add the following file:

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Next, to create a Kafka systemd unit file using the following command:

nano /etc/systemd/system/kafka.service

Add the below content:

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64"
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

Save the files and reload the systemd daemon to apply new changes:

sudo systemctl daemon-reload
sudo systemctl start zookeeper
sudo systemctl start kafka

Step 5. Create a Topic in Kafka.

First, create a topic named “MakeTopic” with a single partition with a single replica:

cd /usr/local/kafka
bin/kafka-maketopics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic MakeTopic

After that, you can see the created topics on Kafka by the running below command:

$ bin/kafka-maketopics.sh --list --zookeeper localhost:2181
MakeTopic

Step 6. Send Messages to Kafka.

Now we run the producer and then type a few messages into the console to send to the server:

$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic MakeTopic

>Welcome to kafka
>This is my first topic
>

Step 7. Using Kafka Consumer.

Apache Kafka also has a command-line consumer to read data from the Kafka cluster and display messages to standard output.

$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic MakeTopic --from-beginning

Welcome to kafka
This is my first topic

Congratulations! You have successfully installed Apache Kafka. Thanks for using this tutorial for installing the latest version of the Apache Kafka on the Debian system. For additional help or useful information, we recommend you check the official Apache Kafka website.