How To Install Apache Kafka on Linux Mint 22

In this tutorial, we will show you how to install Apache Kafka on Linux Mint 22. Apache Kafka stands as a powerful distributed event streaming platform widely adopted by organizations for building high-performance data pipelines, enabling real-time streaming applications, and implementing reliable data integration solutions. Installing Kafka on Linux Mint 22 requires careful attention to detail, but the process becomes straightforward with proper guidance. This comprehensive tutorial walks you through each step of installing and configuring Apache Kafka on your Linux Mint 22 system, ensuring you can harness the full capabilities of this robust messaging system.

Understanding Apache Kafka

Apache Kafka is an open-source distributed event streaming platform originally developed by LinkedIn and later donated to the Apache Software Foundation. It serves as a high-throughput, fault-tolerant messaging system designed to handle real-time data feeds with ease.

What is Apache Kafka?

At its core, Kafka functions as a distributed commit log service, enabling applications to publish and subscribe to streams of records. These records are stored in topics, which are partitioned and replicated across multiple servers for fault tolerance. The architecture consists of several key components:

• Brokers: Servers that store data and serve client requests
• Topics: Categories or feed names to which records are published
• Producers: Applications that send data to Kafka
• Consumers: Applications that process data from Kafka
• ZooKeeper: Coordination service for managing broker metadata

Benefits of Apache Kafka

The popularity of Kafka stems from several technical advantages:

• High throughput: Processes millions of messages per second
• Fault tolerance: Replicates data across multiple nodes
• Durability: Persists messages to disk for reliability
• Scalability: Easily scales horizontally by adding more brokers
• Real-time processing: Enables stream processing with minimal latency
• Ecosystem integration: Works with numerous data systems and frameworks

Prerequisites for Installation

Before proceeding with the installation, ensure your system meets all requirements and has the necessary software components installed.

System Requirements

Apache Kafka isn’t particularly resource-intensive for testing environments, but production deployments require careful hardware planning:

• CPU: 2+ cores recommended (4+ for production)
• RAM: 4GB minimum (8GB+ recommended for production)
• Storage: 20GB+ of free space (SSD preferred for performance)
• Network: Stable connection with sufficient bandwidth
• Linux Mint 22 with current updates installed

Required Software

Java forms the foundation for running Apache Kafka, as the platform is built on the Java Virtual Machine (JVM):

# Update package repository
sudo apt update

# Install OpenJDK 11 or later
sudo apt install -y openjdk-11-jdk

# Verify Java installation
java -version

You should see output confirming Java is installed.

User Permissions

It’s best practice to run Kafka under a dedicated user account rather than root for security purposes:

# Create kafka user and group
sudo adduser --system --group kafka

# Create directories for Kafka data
sudo mkdir -p /opt/kafka /var/lib/kafka/data

# Set proper ownership
sudo chown -R kafka:kafka /opt/kafka /var/lib/kafka

Preparing Your Linux Mint Environment

A well-prepared environment ensures smooth installation and optimal performance for your Kafka deployment.

Updating Your System

Begin by ensuring your Linux Mint system has the latest updates:

sudo apt update
sudo apt upgrade -y

Installing OpenJDK

Kafka requires Java to run. Install OpenJDK, which provides an open-source implementation of the Java Development Kit:

# Install OpenJDK
sudo apt install -y openjdk-11-jdk

# Verify installation
java -version

After installation, set the JAVA_HOME environment variable by adding these lines to your ~/.bashrc file:

echo 'export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64' >> ~/.bashrc
echo 'export PATH=$PATH:$JAVA_HOME/bin' >> ~/.bashrc
source ~/.bashrc

Creating Necessary Directories

Prepare directories for Kafka installation and data storage:

# Create directories
sudo mkdir -p /opt/kafka
sudo mkdir -p /var/lib/kafka/data
sudo mkdir -p /var/log/kafka

# Set permissions
sudo chown -R kafka:kafka /opt/kafka /var/lib/kafka /var/log/kafka

Downloading and Extracting Apache Kafka

With your environment prepared, you can now obtain and extract the Apache Kafka distribution.

Finding the Latest Stable Release

Visit the Apache Kafka website to find the most recent stable release.

Downloading the Kafka Binaries

Use wget to download the Kafka distribution:

# Change to the /tmp directory for downloading
cd /tmp

# Download Kafka (adjust version number as needed)
wget https://dlcdn.apache.org/kafka/3.9.0/kafka-3.9.0-src.tgz

# Verify download integrity (optional but recommended)
wget https://downloads.apache.org/kafka/3.9.0/kafka-3.9.0-src.tgz.sha512
sha512sum -c kafka-3.9.0-src.tgz.sha512

Extracting the Archive

Extract the downloaded tarball and move the contents to the installation directory:

# Extract the archive
tar -xzf kafka-3.9.0-src.tgz

# Move to installation directory
sudo mv kafka-3.9.0-src/* /opt/kafka/

# Set proper ownership
sudo chown -R kafka:kafka /opt/kafka

Configuring Apache Kafka

Proper configuration is crucial for Kafka’s performance, stability, and security.

Understanding Configuration Files

Kafka uses several configuration files located in the /opt/kafka/config directory:

• server.properties: Main broker configuration
• zookeeper.properties: ZooKeeper configuration
• producer.properties: Default producer settings
• consumer.properties: Default consumer settings

Basic Server Configuration

Edit the server.properties file to customize your Kafka broker:

# Broker ID - must be unique in a cluster
broker.id=0

# Listener configuration
listeners=PLAINTEXT://localhost:9092

# Directory where logs are stored
log.dirs=/var/lib/kafka/data

# Default number of partitions for new topics
num.partitions=3

# Log retention settings (default: 168 hours = 7 days)
log.retention.hours=168

# ZooKeeper connection string
zookeeper.connect=localhost:2181

ZooKeeper Configuration

Next, configure ZooKeeper by editing its properties file:

# ZooKeeper data directory
dataDir=/var/lib/zookeeper

# Port to listen on
clientPort=2181

# Disable admin server for simplicity in development
admin.enableServer=false

Network Configuration

For production environments or if you need to access Kafka from other machines, update the network settings in server.properties:

# Listen on all network interfaces
listeners=PLAINTEXT://0.0.0.0:9092

# Advertised hostname (use your machine's actual hostname or IP)
advertised.listeners=PLAINTEXT://your-hostname:9092

Setting Up Kafka as a System Service

To ensure Kafka starts automatically on system boot and can be managed with standard service commands, set up systemd service units.

Creating Systemd Unit Files

First, create a systemd service file for ZooKeeper:

[Unit]
Description=Apache ZooKeeper Server
Documentation=http://kafka.apache.org/documentation.html
Requires=network.target
After=network.target

[Service]
Type=simple
User=kafka
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target

Next, create a service file for Kafka:

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target

Configuring Service Parameters

The service files include several important configurations:

• User: Specifies that services run as the kafka user
• Environment: Sets the JAVA_HOME variable
• ExecStart/ExecStop: Defines start and stop commands
• Restart=on-failure: Automatically restarts the service if it crashes
• Dependencies: Kafka requires ZooKeeper to start first

Enabling and Starting Services

Reload systemd to recognize the new service files:

sudo systemctl daemon-reload

Enable both services to start automatically on boot:

sudo systemctl enable zookeeper.service
sudo systemctl enable kafka.service

Start the services:

sudo systemctl start zookeeper.service
sudo systemctl start kafka.service

Verify that both services are running properly:

sudo systemctl status zookeeper.service
sudo systemctl status kafka.service

Testing Your Kafka Installation

After installation, verify that your Kafka setup works correctly by creating topics and exchanging messages.

Creating Your First Topic

Create a test topic with three partitions and a replication factor of one:

/opt/kafka/bin/kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --replication-factor 1 \
  --partitions 3 \
  --topic test-topic

Producing Test Messages

Use the console producer to send messages to your topic:

/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 \
  --topic test-topic

Consuming Messages

Open a new terminal and use the console consumer to read messages from your topic:

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic test-topic \
  --from-beginning

Basic Kafka Operations

Once Kafka is running, you’ll need to know how to manage topics and monitor your installation.

Managing Topics

Create a new topic:

/opt/kafka/bin/kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --replication-factor 1 \
  --partitions 3 \
  --topic new-topic

List all topics:

/opt/kafka/bin/kafka-topics.sh --list \
  --bootstrap-server localhost:9092

Monitoring Kafka

Check broker performance and metrics:

/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092

Advanced Configuration Options

As your Kafka usage grows, you’ll likely need to optimize performance and enhance security.

Performance Tuning

Fine-tune your Kafka installation by modifying these parameters in server.properties:

# Increase number of network threads
num.network.threads=6

# Increase number of I/O threads
num.io.threads=12

# Increase socket buffer sizes
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576

# Increase request maximum size
message.max.bytes=10485760

# Optimize log flush behavior
log.flush.interval.messages=10000
log.flush.interval.ms=1000

Security Setup

For production environments, implement security measures:

1. Configure SSL/TLS encryption:

listeners=SSL://0.0.0.0:9093
ssl.keystore.location=/path/to/kafka.server.keystore.jks
ssl.keystore.password=keystore-password
ssl.key.password=key-password
ssl.truststore.location=/path/to/kafka.server.truststore.jks
ssl.truststore.password=truststore-password
security.inter.broker.protocol=SSL

Multi-broker Setup

For high availability and increased throughput, set up a multi-broker cluster:

1. Create multiple server.properties files with different broker IDs and ports:

sudo cp /opt/kafka/config/server.properties /opt/kafka/config/server-1.properties
sudo cp /opt/kafka/config/server.properties /opt/kafka/config/server-2.properties

Troubleshooting Common Issues

Even careful installations can encounter problems. Here are solutions to common Kafka issues.

Connection Problems

If clients cannot connect to Kafka:

• • Verify Kafka is running:

sudo systemctl status kafka

• • Check listener configuration in server.properties:

grep listeners /opt/kafka/config/server.properties

• • Ensure ports are open in the firewall:

sudo ufw status
sudo ufw allow 9092/tcp  # If needed

Performance Issues

If Kafka performs poorly:

• • Check disk I/O performance:

sudo apt install -y iotop
sudo iotop -o

• • Monitor CPU and memory usage:

top

Log-related Problems

Issues with log storage and management:

• • Check available disk space:

df -h

• • Adjust log retention if space is low:

/opt/kafka/bin/kafka-configs.sh --alter \
  --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name your-topic \
  --add-config retention.ms=86400000

Congratulations! You have successfully installed Kafka. Thanks for using this tutorial for installing Apache Kafka on Linux Mint 22 system. For additional help or useful information, we recommend you check the official Apache Kafka website.