Embedded Cluster

This article describes how to set up and connect to the embedded cluster with KaDeck.

What is the embedded cluster functionality?

With KaDeck 1.4, the "embedded cluster" was introduced. This new functionality for fast local testing allows you to start up an Apache Kafka broker (and Zookeeper) directly in KaDeck. By "instrumenting" a standard Apache Kafka installation, no flexibility is lost. It is still possible to manually configure all broker settings (Kafka and Zookeeper) and to choose the Apache Kafka version you want.

Setup

Head over to the Apache Kafka project page and download the binary version of Apache Kafka. Un-tar the downloaded folder and move it to a location that is accessible for KaDeck. 

For Windows users: as there are some constraints on path lengths on Windows which can result in strange behavior and weird error messages, we recommend placing the folders content close to the root of your hard drive (e.g. C:/kafka/ so that the path to /bin is C:/kafka/bin).

Next, start KaDeck and enter the path to your extracted folder's content (e.g. C:/kafka/) and click "Start".

Fig. 1 Embedded cluster card in online state.

Click on "Start" and wait several seconds. The starting process can take a while, depending on your machine.

Once the cluster is running, you can connect using a new server configuration with the local broker address (usually 127.0.0.1:9092). 

If an error occurred, check the log files in your folder (e.g. C:/kafka/zookeeper.out).

Note: Clicking on "Start" does kill any previous Kafka broker and Zookeeper instances first, deletes the data dir and log dir and regularly starts both servers.

FAQ

There are no topics (or data) after restarting my embedded cluster.

KaDeck deletes all information from previous sessions once the embedded cluster is started. This behavior guarantees that you are starting with a clean environment every time and the cluster does not get into an error state.

I have closed KaDeck without stopping the cluster.

The cluster is still running in the background. Open KaDeck again and click on "Start" on the embedded cluster card. This will stop any instances (if existent) and start the server again. You can then stop the server regularly by clicking "Stop". 

I want to change the Kafka or Zookeeper settings

Edit the server.properties and zookeeper.properties inside the /config folder to customize the settings.