Modern Apache Kafka for Data Engineers

Modern Apache Kafka for Data Engineers: Design, Stream, and Scale Real-Time Data Systems That Power Analytics, Machine Learning, and Business Intelligence
By Leon Amsel
How do the world's top data engineering teams turn raw, continuous data into real-time intelligence that drives analytics, machine learning, and business decisions? They master Apache Kafka, and this book shows you exactly how.
Today's organizations run on data in motion. But designing reliable, scalable, and cost-efficient Kafka-based pipelines isn't just about knowing APIs, it's about mastering system design, automation, resilience, and operational best practices. Modern Apache Kafka for Data Engineers gives you the complete, production-ready playbook to move from concepts to real-world deployment.
Leon Amsel distills years of experience building and running high-throughput data systems into clear, repeatable strategies. Through practical labs, proven patterns, and detailed configuration examples, you'll learn how to design streaming architectures that deliver consistent performance, whether you're running on-prem, in the cloud, or across hybrid environments.
You'll learn how to:

• Build production-grade Kafka clusters with KRaft mode, Kubernetes, and managed cloud platforms.
• Design topics, partitions, and replication strategies for performance, reliability, and scale.
• Implement schema evolution and serialization with Avro, Protobuf, and the Schema Registry.
• Integrate Kafka with external systems using connectors for JDBC, S3, Elasticsearch, and more.
• Create stream-processing and real-time analytics pipelines with Kafka Streams and ksqlDB.
• Deploy, monitor, and secure clusters with observability stacks, ACLs, and encryption.
• Operate at scale through rebalancing, disaster recovery, and cost optimization techniques.