Apache Kafka has become the backbone of modern data infrastructure, powering real-time streaming applications for companies worldwide. This distributed event streaming platform excels at handling high-throughput data pipelines, messaging systems, and stream processing workloads. Whether you’re building a real-time analytics dashboard, implementing event-driven microservices, or creating a robust messaging system, Kafka delivers the performance and reliability your applications demand.

This comprehensive guide walks you through installing Apache Kafka on Debian 13, covering both the modern KRaft mode (which eliminates ZooKeeper dependency) and the traditional ZooKeeper-based setup. You’ll learn everything from initial system preparation to testing your first producer and consumer, complete with troubleshooting tips and security considerations. By the end of this tutorial, you’ll have a fully functional Kafka installation ready for development or production deployment.

Understanding Apache Kafka Architecture

Before diving into installation, understanding Kafka’s core architecture helps you make informed configuration decisions.

Kafka operates through several key components working together. Brokers are servers that store and serve data, handling read and write requests from clients. Topics organize messages into categories, similar to database tables or message queues. Producers are applications that publish data to topics, while consumers read data from topics for processing.

The platform now supports two operational modes. KRaft mode represents the modern approach, using an internal consensus protocol that eliminates the need for external ZooKeeper coordination. Traditional mode relies on ZooKeeper for cluster metadata management and leader election. KRaft simplifies deployment and improves scalability, making it the recommended choice for new installations.

Kafka’s architecture delivers exceptional throughput and low latency. The system can handle millions of messages per second while maintaining millisecond-level response times, making it ideal for real-time data processing scenarios.

Prerequisites and System Requirements

Proper preparation ensures smooth installation. Your Debian 13 system needs adequate resources to run Kafka effectively.

Allocate at least 2GB of RAM, though 4GB is recommended for production environments. Reserve 10GB of free disk space for Kafka binaries, logs, and data storage. You’ll need root or sudo access to install packages and configure system services.

Kafka requires Java Development Kit (JDK) version 11 or higher to run. Network configuration matters too—ensure ports 9092 (Kafka) and 2181 (ZooKeeper, if used) are available. Consider firewall rules if you plan to access Kafka from remote machines.

Security best practices dictate running Kafka as a dedicated non-root user. This isolation limits potential damage from security vulnerabilities and follows the principle of least privilege.

Start by updating your system packages:

sudo apt update && sudo apt upgrade -y

This ensures you’re working with the latest security patches and package versions.

Step 1: Install Java Development Kit

Java powers Kafka’s runtime environment. Installing the correct version is your first technical step.

Debian 13 repositories include OpenJDK packages that work perfectly with Kafka. Install both the Java Runtime Environment (JRE) and Development Kit (JDK):

sudo apt install default-jre default-jdk -y

Verify your Java installation succeeded:

java -version

You should see output indicating Java version 11 or higher. The command displays the installed version, runtime environment, and build information.

Set the JAVA_HOME environment variable for system-wide availability. Edit the environment file:

sudo nano /etc/environment

Add this line at the end:

JAVA_HOME="/usr/lib/jvm/default-java"

Load the new environment variable:

source /etc/environment

Verify JAVA_HOME is set correctly:

echo $JAVA_HOME

If Java tools like javac and jar are accessible, your installation is complete and properly configured.

Step 2: Create Dedicated Kafka User

Running services as non-root users enhances security significantly. Create a dedicated user for Kafka operations.

Execute this command to create the kafka user with a home directory:

sudo useradd -r -m -U -d /home/kafka -s /bin/bash kafka

The flags specify a system user (-r), create home directory (-m), create a group with the same name (-U), set home directory path (-d), and assign bash shell (-s).

Set a password for the kafka user:

sudo passwd kafka

Add the kafka user to the sudo group for administrative tasks:

sudo adduser kafka sudo

Switch to the kafka user to perform the remaining installation steps:

su -l kafka

This approach isolates Kafka processes from the root account. If Kafka encounters security issues or bugs, the impact remains limited to the kafka user’s permissions rather than compromising the entire system.

Create a working directory structure for organized file management:

mkdir -p ~/Downloads

Step 3: Download Apache Kafka

Obtaining Kafka binaries from official sources ensures you get authentic, unmodified software.

Visit the Apache Kafka downloads page to identify the latest stable release. At the time of writing, Kafka 3.x versions offer excellent stability and features. Install curl and wget if they’re not already on your system:

sudo apt-get install curl wget -y

Download the Kafka binary distribution. Replace the version numbers with the current stable release:

curl "https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz" -o ~/Downloads/kafka.tgz

The filename indicates Kafka version 3.6.1 compiled for Scala 2.13. Kafka includes Scala binaries, so you don’t need to install Scala separately.

For enhanced security, verify the download’s integrity using checksums. The Apache downloads page provides SHA512 checksums alongside each release. Download the checksum file:

curl "https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz.sha512" -o ~/Downloads/kafka.tgz.sha512

Verify the checksum matches:

sha512sum ~/Downloads/kafka.tgz
cat ~/Downloads/kafka.tgz.sha512

Compare the output from both commands. Matching checksums confirm your download hasn’t been tampered with.

List the downloaded file to confirm its presence:

ls -lh ~/Downloads/

Step 4: Extract and Install Kafka

Extract the Kafka archive to create your installation directory.

Create a dedicated directory for Kafka:

sudo mkdir /opt/kafka

Extract the downloaded archive directly to the installation directory:

sudo tar -xvzf ~/Downloads/kafka.tgz -C /opt/kafka --strip-components=1

The –strip-components=1 flag removes the top-level directory from the archive, placing Kafka files directly in /opt/kafka rather than creating a subdirectory.

Verify the extraction completed successfully:

ls -1 /opt/kafka

You should see directories including bin (executables), config (configuration files), libs (Java libraries), and licenses.

Set proper ownership so the kafka user can access all files:

sudo chown -R kafka:kafka /opt/kafka

Clean up the downloaded archive to free disk space:

rm ~/Downloads/kafka.tgz

Understanding the directory structure helps you navigate Kafka’s organization. The bin directory contains shell scripts for starting services and managing topics. The config directory holds configuration files for different deployment scenarios. The libs directory includes all Java dependencies Kafka needs to run.

Step 5: Configure Apache Kafka (KRaft Mode)

KRaft mode represents Kafka’s future. It eliminates ZooKeeper dependency, simplifying architecture and improving scalability.

Generate a unique cluster UUID that identifies your Kafka cluster:

/opt/kafka/bin/kafka-storage.sh random-uuid

Save this UUID—you’ll need it for formatting storage. It looks like a standard UUID format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.

Edit the KRaft configuration file:

nano /opt/kafka/config/kraft/server.properties

Configure these essential parameters:

node.id=1
log.dirs=/var/lib/kafka-logs
listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://localhost:9092

The node.id uniquely identifies this broker in the cluster. Use sequential numbers starting from 1 for multi-broker setups. The log.dirs parameter specifies where Kafka stores message data. The listeners parameter defines network interfaces and ports Kafka binds to. The advertised.listeners parameter tells clients how to reach this broker.

Create the log directory with proper permissions:

sudo mkdir -p /var/lib/kafka-logs
sudo chown -R kafka:kafka /var/lib/kafka-logs

Format the storage directory using your generated UUID:

/opt/kafka/bin/kafka-storage.sh format -t YOUR_UUID_HERE -c /opt/kafka/config/kraft/server.properties

Replace YOUR_UUID_HERE with the UUID you generated earlier. This command initializes the storage directory structure Kafka needs.

KRaft mode offers several advantages. Startup time improves because Kafka doesn’t wait for ZooKeeper synchronization. Operational complexity decreases with fewer moving parts. Scalability increases as cluster metadata operations become more efficient.

Step 6: Configure Apache Kafka (Traditional ZooKeeper Mode)

Some scenarios still require ZooKeeper mode. Legacy systems, specific compatibility requirements, or organizational standards might dictate this approach.

First, configure ZooKeeper. Edit the configuration file:

nano /opt/kafka/config/zookeeper.properties

Set these parameters:

dataDir=/var/lib/zookeeper
clientPort=2181
maxClientCnxns=0

The dataDir specifies where ZooKeeper stores its data. The clientPort defines the port for client connections. The maxClientCnxns setting of 0 removes connection limits.

Create the ZooKeeper data directory:

sudo mkdir -p /var/lib/zookeeper
sudo chown -R kafka:kafka /var/lib/zookeeper

Now configure the Kafka broker. Edit the server configuration:

nano /opt/kafka/config/server.properties

Configure these critical settings:

broker.id=0
log.dirs=/var/lib/kafka-logs
listeners=PLAINTEXT://localhost:9092
zookeeper.connect=localhost:2181

The broker.id must be unique in a multi-broker cluster. The zookeeper.connect parameter tells Kafka where to find ZooKeeper.

Create the Kafka log directory:

sudo mkdir -p /var/lib/kafka-logs
sudo chown -R kafka:kafka /var/lib/kafka-logs

ZooKeeper mode requires careful startup sequencing. ZooKeeper must start and stabilize before Kafka begins. This dependency adds operational complexity but remains well-tested and stable.

Step 7: Create Systemd Service Files

Systemd integration enables automatic startup, process monitoring, and centralized logging. This professional approach suits production deployments.

For KRaft Mode

Create the Kafka service file:

sudo nano /etc/systemd/system/kafka.service

Add this configuration:

[Unit]
Description=Apache Kafka Server (KRaft Mode)
Documentation=https://kafka.apache.org/documentation/
After=network.target

[Service]
Type=simple
User=kafka
Group=kafka
Environment="JAVA_HOME=/usr/lib/jvm/default-java"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

For ZooKeeper Mode

Create the ZooKeeper service first:

sudo nano /etc/systemd/system/zookeeper.service

Add this content:

[Unit]
Description=Apache ZooKeeper Server
Documentation=https://zookeeper.apache.org
After=network.target

[Service]
Type=simple
User=kafka
Group=kafka
Environment="JAVA_HOME=/usr/lib/jvm/default-java"
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Create the Kafka service file:

sudo nano /etc/systemd/system/kafka.service

Add this configuration:

[Unit]
Description=Apache Kafka Server
Documentation=https://kafka.apache.org/documentation/
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
Group=kafka
Environment="JAVA_HOME=/usr/lib/jvm/default-java"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

The Requires and After directives ensure ZooKeeper starts before Kafka and that Kafka stops if ZooKeeper fails.

Step 8: Start and Enable Kafka Services

With systemd files configured, activate your Kafka installation.

Reload systemd to recognize the new service files:

sudo systemctl daemon-reload

For KRaft Mode

Start the Kafka service:

sudo systemctl start kafka

Enable automatic startup at boot:

sudo systemctl enable kafka

For ZooKeeper Mode

Start ZooKeeper first:

sudo systemctl start zookeeper

Enable ZooKeeper to start at boot:

sudo systemctl enable zookeeper

Wait 10-15 seconds for ZooKeeper to initialize completely. Then start Kafka:

sudo systemctl start kafka

Enable Kafka for automatic startup:

sudo systemctl enable kafka

Check the service status:

sudo systemctl status kafka

A successful start shows “active (running)” in green text. The status output displays recent log entries showing Kafka’s initialization process.

Typical startup takes 5-15 seconds depending on your hardware. Kafka performs several initialization tasks including loading configuration, initializing storage, and establishing network listeners.

Step 9: Verify Kafka Installation

Verification confirms Kafka runs correctly and accepts connections.

Check that Kafka processes are running:

ps -ef | grep kafka

You should see Java processes with kafka in their command line. The output shows process IDs, user ownership, and full command arguments.

Verify port binding:

sudo ss -tuln | grep 9092

This command shows Kafka listening on port 9092. The output indicates the protocol (TCP), listening address, and port number.

Examine Kafka logs for any errors:

sudo journalctl -u kafka -n 50

Scroll through the last 50 log entries. Look for successful startup messages indicating Kafka has completed initialization. Error messages appear clearly marked and often include stack traces.

For ZooKeeper mode, verify ZooKeeper is running:

sudo systemctl status zookeeper

Test network connectivity to Kafka:

telnet localhost 9092

A successful connection proves Kafka accepts TCP connections. Press Ctrl+] and type “quit” to exit telnet.

Common startup issues include Java version mismatches, permission errors on log directories, or port conflicts. Log files provide detailed error information to diagnose problems.

Step 10: Create and Test Kafka Topic

Topics organize messages in Kafka. Creating a test topic validates your installation’s functionality.

Create a simple test topic:

/opt/kafka/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

The command creates a topic named “test-topic” with one partition and one replica. Partitions enable parallel processing, while replication provides fault tolerance.

List all topics to confirm creation:

/opt/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Your test-topic should appear in the output.

Describe topic details:

/opt/kafka/bin/kafka-topics.sh --describe --topic test-topic --bootstrap-server localhost:9092

The output shows partition count, replication factor, leader broker, and replica locations. This information helps you understand topic configuration and cluster state.

Topic naming follows best practices. Use lowercase letters, numbers, hyphens, and underscores. Descriptive names like “user-events” or “order-processing” improve maintainability.

Delete the test topic if needed:

/opt/kafka/bin/kafka-topics.sh --delete --topic test-topic --bootstrap-server localhost:9092

Note that older Kafka versions using ZooKeeper mode required the –zookeeper flag instead of –bootstrap-server. Modern versions standardize on –bootstrap-server for consistency.

Step 11: Test Kafka Producer

Producers write data to Kafka topics. Testing producer functionality verifies the complete write path.

Start the console producer:

/opt/kafka/bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092

The console waits for input. Type messages and press Enter after each:

Hello Kafka
This is a test message
Apache Kafka on Debian 13 works perfectly

Each line becomes a separate message in the topic. The producer sends messages immediately when you press Enter.

Exit the producer with Ctrl+C.

Producers support configuration options affecting reliability and performance. The acks parameter controls acknowledgment requirements. Setting acks=all ensures all replicas receive messages before acknowledging, maximizing durability at the cost of latency.

Test with producer properties:

/opt/kafka/bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092 --producer-property acks=all

Connection issues typically manifest as timeout errors or “broker not available” messages. Verify Kafka is running and listening on the correct port. Check firewall rules if connecting remotely.

Step 12: Test Kafka Consumer

Consumers read data from topics. Testing consumer functionality validates the complete read path.

Start the console consumer:

/opt/kafka/bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092

The –from-beginning flag instructs the consumer to read all messages from the topic’s start. Without this flag, consumers only receive new messages sent after they connect.

You should see the messages you sent with the producer:

Hello Kafka
This is a test message
Apache Kafka on Debian 13 works perfectly

Messages appear in the order they were written. Kafka guarantees ordering within each partition.

Exit the consumer with Ctrl+C.

Consumer groups enable parallel processing. Multiple consumers in the same group divide partition assignments, scaling read throughput. List consumer groups:

/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

View consumer lag (messages waiting to be read):

/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group YOUR_GROUP_NAME

Consumer lag indicates processing backlog. High lag suggests consumers can’t keep up with production rates, signaling a need for scaling or optimization.

Firewall Configuration for External Access

Remote access requires firewall configuration. Opening appropriate ports enables external clients to connect.

If using UFW (Uncomplicated Firewall), open the Kafka port:

sudo ufw allow 9092/tcp

For ZooKeeper mode, also open ZooKeeper’s port:

sudo ufw allow 2181/tcp

Reload the firewall to apply changes:

sudo ufw reload

Configure advertised.listeners for external clients. Edit your server properties file and update:

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://your-server-ip:9092

Replace your-server-ip with your server’s public IP address or domain name. The listeners parameter accepts connections on all interfaces (0.0.0.0), while advertised.listeners tells clients the correct address to use.

Restart Kafka after configuration changes:

sudo systemctl restart kafka

Test external connectivity from a remote machine:

telnet your-server-ip 9092

Production environments should implement additional security. Exposing Kafka directly to the internet without authentication creates serious security risks. Consider using VPNs, security groups, or Kafka’s built-in authentication mechanisms.

Basic Security Configuration

Security protects your Kafka deployment from unauthorized access. While comprehensive security setup requires extensive configuration, understanding the basics proves valuable.

Kafka supports SASL (Simple Authentication and Security Layer) for authentication. SASL mechanisms include PLAIN, SCRAM, and Kerberos. SCRAM provides strong authentication with encrypted passwords stored in Kafka.

SSL/TLS encryption protects data in transit. Enabling SSL requires generating certificates, configuring keystores and truststores, and updating listener configurations. The encrypted channel prevents network eavesdropping and tampering.

Inter-broker security ensures secure communication between Kafka brokers in multi-broker clusters. Configure separate listeners for client and inter-broker traffic with appropriate security settings.

Access Control Lists (ACLs) provide topic-level permissions. Grant specific users or applications permission to read from or write to particular topics. This fine-grained control follows the principle of least privilege.

Creating a JAAS (Java Authentication and Authorization Service) configuration file enables SASL authentication:

sudo nano /opt/kafka/config/kafka_server_jaas.conf

Update server.properties to reference the JAAS file and enable security features. Full security implementation demands careful planning and testing. Improperly configured security can lock you out of your own cluster or create vulnerabilities.

Monitoring and Performance Considerations

Monitoring reveals system health and performance characteristics. Production deployments require observability to maintain reliability.

Key metrics include broker CPU and memory usage, producer and consumer lag, topic-level throughput, and disk usage. These indicators help you identify bottlenecks and capacity constraints before they impact applications.

Prometheus with Kafka Exporter provides robust monitoring. The exporter scrapes JMX metrics from Kafka and exposes them in Prometheus format. Grafana dashboards visualize these metrics with graphs, charts, and alerts.

JMX (Java Management Extensions) exposes internal Kafka metrics. Enable JMX by setting environment variables in your systemd service file:

Environment="KAFKA_JMX_OPTS=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

Performance tuning adjusts Kafka’s behavior for your workload. Increase heap size for high-throughput scenarios by modifying KAFKA_HEAP_OPTS. Adjust batch.size and linger.ms for producers to balance latency and throughput.

Common Troubleshooting Issues

Even careful installations encounter problems. Understanding common issues accelerates resolution.

Kafka service fails to start: Check Java installation completeness. Verify log directory permissions allow kafka user write access. Examine journalctl output for specific error messages. Java version mismatches or missing JAVA_HOME variables frequently cause startup failures.

Connection refused errors: Verify Kafka is running and listening on the correct port. Check firewall rules permit traffic on port 9092. Ensure listeners configuration matches your connection attempts. Network interface binding to localhost prevents external connections.

Out of memory errors: Kafka’s default heap size may prove insufficient. Increase memory allocation by setting KAFKA_HEAP_OPTS in your systemd service file. Production deployments typically allocate 4-8GB heap size depending on workload.

ZooKeeper connection issues: Confirm ZooKeeper service runs before starting Kafka. Verify the zookeeper.connect parameter points to the correct address and port. Check ZooKeeper logs for initialization problems. ZooKeeper requires a few seconds to stabilize after starting.

Topic creation failures: Ensure brokers are running and reachable. Check that you have sufficient replicas available for your replication factor. Verify no permission issues prevent directory creation.

Meta.properties conflicts: Cluster ID mismatches prevent startup. Delete meta.properties files from your log directories and reformat storage if switching between KRaft and ZooKeeper modes or changing cluster configurations.

Log files provide crucial diagnostic information. Check systemd journal logs:

sudo journalctl -u kafka -f

The -f flag follows log output in real-time. Kafka’s own log files reside in /opt/kafka/logs/ and contain detailed debugging information.

Enable debug logging by modifying log4j.properties in the config directory. Change log levels from INFO to DEBUG for verbose output helping diagnose complex issues.

Congratulations! You have successfully installed Apache Kafka. Thanks for using this tutorial to install the latest version of Apache Kafka on Debian 13 “Trixie” system. For additional help or useful information, we recommend you check the official Apache website.