What are callback topics and when you would use them – Asynchronous design patterns

By | November 29, 2020

An explanation of when to use callback topics

Update: 18/01/2021 – Since publishing this post I have since uploaded a YouTube Video and have now updated this blog post with more content.
I have added a link to the YouTube post at the end.

Callbacks topics are a handy tool in an architect’s tool belt. Why? Well, let us discuss. In an event-driven architecture, one of the main drivers is to allow plugin play. What is plugin play, this is when you don’t need to keep changing existing applications when you need to add a new process or additional to notify other processes/systems. If you have a function which processes items what you could do is to have the processor publish state changes to the topic.

A topic is a Kafka naming convention. A topic is a log of messages, an event stream, a queue of events/messages

Let’s say you are designing a solution where multiple publishers will trigger asynchronous processing of a message, and you require the processor (the consumer) of the triggering events to publish an event with the results of the processing to a queue which the publishing process system can handle the result.

You could publish all the result irrelevant of which originating system or process requires the result; this would result in all the subscribers (consumers) of the results to have to read and filter out messages which could be costly. As shown in the below image.

One result topic (event log/stream) for all processes

Or you could publish each message to a specific topic which for each process which requires the results, this in my view is the preferred solution.

However, you could add logic and hard code the processor to publish to the originating publishers preferred topic. However, this implementation is one which I believe could have long term negative impact on the development process.

  1. The processor would have to know the broader context of the message it is processing, which in my view could put too much responsibility onto to the function which is processing the message
  2. How would a developer decide which topic to publish the results to; by implementing an if statement, a switch statement, using delegates.
  3. Do you keep changing the service when new processes need to know about the results, which will involve full development cycles to modify, test and update a stable service
  4. Hard coding topics to publish to would couple the services together.

This is where the callback topic comes into play.

Each process has its own callback topic which a result handler will consume

How to implement callback topics

First, you need to define a schema for your event messages. In that schema, you will need to define a few “header” properties which would tell the processing service were to publish the result events to. The two which I add are:

  1. The callback topic name which the processor/consumer will publish the results to.
  2. The callback reference Id which the consumer of the outcome event will use to handle the reconciliation of the results and the item it was processing when it triggered the processor to perform its actions.
{
  "headers": {
    "callbackReferenceId": "4a97d248....",
    "callbackTopic": "Process 1 Callback Topic"
  },
  "body": {....}
}

The second thing you do is implement the logic in the processor/consumer which will after processing check to see if the callback topic properties are not null and if they are not null will wrap the processing results in a message along with a “header” containing

  1. The callback reference id
  2. The name of the operation or the event. The outcome of the operation. THis is useful if the topic which the message will be published to also contain messages from other services & operations. This is normally useful when you are using the orchestration pattern.
{
  "headers": {
    "callbackReferenceId": "4a97d248....",
    "operation": "process.operation/event/outcome",
    ....
  },
  "body": {
    "successful": true,
    ....
  }
}

And lastly, you have the consumer of the result events consuming the callback topic.

Of course, you must ensure that the call back topic is created before publishing or subscribing to it. If you have implemented security where you need to explicitly define at the (Kafka/application) user level which users can publish and consume which topics, you will, of course, need to ensure that actions are taken.

Summary

Having designed solutions which have been running in production for some time and which have required to have new processes plugged on, the only thing we have had to do is assign the processor’s Kafka user another topic it can publish to. Therefore not having to touch a stable service.

Leave a Reply