Back in 2021, when I first came across Kafka, I remember the DevOps engineer in my team using terms like Zookeeper, broker configs, etc on our team standup calls. I remember not caring about those terms, and simply focusing on learning about the producer, partitions, topics and consumer groups, and how they could be used in the product my team was developing. While platforms like Kafka were built to abstract certain aspects of distributed computing (replication, consistency) while storing & processing logs, it's a pity how so many of us miss out on knowing the amazing engineering that went behind to build the different parts of a platform such as this one. 4 years later, I'm hungry enough to reverse engineer one of my favourite distributed platforms - Kafka! What did the Zookeeper do? To quote the 1st Kafka paper from 2011, Kafka uses Zookeeper for the following tasks: (1) detecting the addition and the removal of brokers and consumers, (2) triggering a rebalance ...