Tag Archives: Apache Kafka

Part 2: Sagas, Fan-In, and Correlation: Solving the Hard Problems of Eventual Consistency in EDA

Choosing event-driven architecture means choosing eventual consistency. The only question is how well you manage it. When payment succeeds but inventory reservation fails, do you refund automatically or let a customer stare at a half-completed order? This guide covers Saga compensation patterns, the callback topic debate (and why it’s really async RPC over event infrastructure), fan-out vs fan-in complexity, and three concrete approaches to multi-topic correlation — with production code in Python, Java, and AWS.

Before You Adopt Event-Driven Architecture: Prerequisites, Red Flags, and Partition Strategy

Most teams adopting event-driven architecture never stop to understand what it actually involves. Before committing to the operational overhead that transforms a three-component system into fifteen, verify you actually have the problems EDA exists to solve. This guide provides a decision framework, prerequisites checklist, and a deep dive into partition strategy — the most consequential design decision you’ll make in EDA.

An introduction to Apache Kafka & Amazon MSK (Managed Kafka Service)

Explore the fundamentals of Apache Kafka and Amazon MSK, the cutting-edge solutions for real-time data streaming and processing. Designed for developers and architects, this guide introduces you to the high-throughput, low-latency capabilities of Kafka and the ease of management provided by Amazon MSK. Learn about Kafka’s publisher-subscriber model, efficient data handling, and the seamless integration with AWS services. Discover how these technologies facilitate robust, scalable communication between data producers and consumers, ensuring high availability, security, and seamless data flow across distributed systems.

Why are asynchronous Event-driven microservices complex compared?

An example of the additional complexities which event-driven architecture brings. In this example, I use a combination of Apache Kafka (AWS MKS), Oracle, and Amazon Data Migration Service (AWA DMS) to change an existing synchronous request-driven microservice to a asynchronous event-driven solution.

How to delay reprocessing when consuming from a Apache Kafka (retry) topic

Have you watched the video How to handle message retries & failures in event driven-systems? Handling retires with Kafka? , but you need more information. Hopefully this post helps. Let’s say you have a consumer, a retry consumer, subscribing to a retry topic. When the retry consumer consumes the retry topic and retrieves message/s, the consumer must check… Read More »

Why not having a Schema registry Can Be a Fatal Mistake in Event-Driven Architecture

How to ensure events are processed in order when retrying events out of sequence (when using event platforms like Apache Kafka)

In the post, How to handle retry messages and failures when using event streaming platforms like Apache Kafka, I walked through how to retry events when using Apache Kafka. Now in this post, I will discuss an approach you can implement to ensure processors process events in order even when you’ve implemented a retry solution. YouTube video After… Read More »