Hello World with a basic Kafka Producer and Consumer

Hello World with a basic Kafka Producer and Consumer. It is assumed that you know Kafka terminology. There is a lot to learn about Kafka, but this starter is as simple as it can get with Zookeeper, Kafka and Java based producer/consumer.

Download and install Kafka 2.12. Install in this case is just unzip. Open two console windows to your Kafka directory (named such as kafka_2.12-2.3.0)


Play around with Kafka using CLI commands to check if all is good and also create the topic named testtopic with a replication factor of 1 (limited to 1 since we started only 1 broker) and partition size of 2 (though in our example the 2nd partition is not used since the message key is hardcoded. Left as an exercise to generate keys for each message.


And now the Java code to publish and subscribe. I used Eclipse STS and you can run the producer and consumer from the IDE. In the producer console type in a string and hit enter. The text will be published to a test topic. The consumer will continue to run forever and pick up any messages published and print to console. Type in “exit” to stop the publisher (or you can kill it). Consumer can be killed when you are done testing.

To run the java producer (after you build the mvn artifact using mvn clean package)

To run the java consumer

Type something into the console for the producer and it should echo back in the consumer console window … see below

Code is at https://github.com/thomasma/hellokafka

Note: This is a very basic (hello world) example of using Kafka just using Kafka APIs (no Spring). Using Kafka in a real app needs tons of more thought. Everything from choosing cluster architecture, to how you assign message keys, to how design right partition strategy, data security, high availability architecture, etc. are to be thought through.