Microservices - Transactional Messaging and handling 2 phase commit problem

 

In a microservices architecture, when data is updated in one microservice, another service may also require to reflect the updated state or the services may be involved in a SAGA pattern for completing a transactional outcome.

But updating the data in the database and publishing the message is a 2 phase commit problem. The database update could be successful whereas the publishing of the message could fail or vice versa. If one think of storing and republishing the messages only on failure, then this can result in messages going out of order.

The typical way to handle this issue is to use transactional outbox pattern, i.e, store the message to be published in a table every time the data is updated. This would mean that the entire operation can be committed in a single transaction and no 2 phase commit problem is involved.

Now, this message needs to be published from the transactional outbox table to the mesaging broker for the other microservices to take corresponding actions. This can be done via one of the following approaches:

1. Polling publisher: There can be a Polling publisher which frequently checks this transactional outbox table and picks the data in the timeCreated order and publishes the messages in the same order. If the messaging broker is down or not reachable while trying to publish the message, the Polling publisher can do finite retries and quit for the next iteration to publish the message. The message publishing can get delayed slightly with this approach based on the frequency of the Polling publisher. So the frequency of the Polling publisher needs to be adjusted based on the use case. The polling publisher need to delete the data from Transactional Outbox table once the message is published.

2. Transaction Log tailing: There are various connectors available with the messaging brokers like kafka for tailing a source database and publishing the messages and even to a target sink database. Such connectors can be used for tailing the transactional outbox table and the connector can be used for publishing the messages. This approach is reliable but is database specific and a bit complex. So if the minor delay is fine for the use case, polling publisher could be the go to solution.

But Transaction Log tailing may be a requirement for a NOSQL database to handle this scenario. The NOSQL database typically will not be having a transaction and hence to store the messages in a single transaction to the database, the generated message may also be required to be stored in the same table as the one in which domain object is stored. Now the challenge will be to identify the messages that needs to be published for a polling publisher. Using transactional log tailing will help to generate messages for only the newly created/updated data eliminating this challenge. 

Connector frameworks:

Debezium

Linkedin Databus

Dyanamo DB streams

Confluent connectors