KAFKA Build utilities with the Kafka server

Nowadays, quite a few data architectures involve both a database and Apache Kafka, which is a distributed streaming platform and the subject of this tutorial. You can also find Kafka described as a publish-subscribe message system, which is a fancy way of saying the same thing as it being a distributed streaming platform. As Kafka is optimised for speed, it doesn’t offer too many facilities to users or administrators – like the ones offered by relational databases – and Kafka commands are long and difficult to remember, worried yet?

Don’t worry as here we’ll look at the basics of Kafka, including plenty of the commands and two Go utilities for working with it.

Installation

We’ll use Kafka from a Docker image. The main reason for this is that Kafka has many dependencies and parameters that complicate the installation process – but Docker images come ready to run and include all required dependencies.

For this tutorial we’ll use a Kafka Docker image offered by Landoop. This is one of the best that you can find because it enables you to work with Kafka as soon as you download it! Other Kafka Docker images, including the one offered by Confluent, require many customisations that demand time, which might discourage amateur users.