In this article, I will share my thoughts on using centralised versus decentralised orchestration when building event-driven serverless systems. I have over 15 years of experience in designing and developing distributed event-driven systems. I have spent years migrating from monolithic systems to microservices and now to serverless-powered systems.
Event-driven architecture can help build highly loosely coupled, fault-tolerant, scalable, and easily-extendable systems, allowing new functionality to be brought to market more quickly.
Event-driven architectures’ natural pattern choreography enables services or business domain functions to subscribe to events streams or topics without requiring modifications to the upstream processor or to a central orchestration service.
Benefits of Choreography
Therefore Choreography :
- Reduces cross-coordination and dependencies between development teams,
- Reduces overall development efforts and waiting time,
- Reduces time to market by increasing agility and team velocity,
- Increases the ability to handle unpredictable traffic by allowing individual services to scale independently and process events at their own pace.
With the rise of serverless and cell-based architecture, each cell is an individual single-responsibility function, such as an AWS Lambda function that should be independently deployable, testable, and replaceable.
Therefore, it is important to understand the trade-offs between centralised orchestration and decentralised choreography when implementing a business process with serverless functions.
Disadvantages of Choreography
Because choreography allows for loose coupling and increased agility, it can increase complexity in maintaining observability and monitoring as more single-responsibility functions are added over time.
Making it challenging to understand the state of the architecture and maintain architectural documentation and knowledge.
Using both Choreography and Orchestration in serverless systems
Therefore, based on my experience, I recommend you to use a centralised orchestration service like Amazon Step Functions to orchestrate processing at the micro-level within a bounded context. Then, the centralised orchestration service can publish events to event streams that are exposed outside of the bounded context at specific milestones.
We can then at the macro level use the choreography pattern. Enabling processors across bounded contexts and business domains to subscribe to and consume those events independently.
Using this approach has allowed my teams to maintain the level of observability & monitoring capabilities which we are used to when using centralised orchestration.
Whilst still providing the benefits of event-driven architecture and decentralised choreography that allows my teams and business units to deliver independently and increase time to market.
The key takeaway is that it is not a question of orchestration OR choreography. It’s an AND question; finding the right balance between the two different patterns is key to building a successful system using event-driven and serverless architecture.