Microservices - Transactional Messaging and handling 2 phase commit problem

 

In a microservices architecture, when data is updated in one microservice, another service may also require to reflect the updated state or the services may be involved in a SAGA pattern for completing a transactional outcome.

But updating the data in the database and publishing the message is a 2 phase commit problem. The database update could be successful whereas the publishing of the message could fail or vice versa. If one think of storing and republishing the messages only on failure, then this can result in messages going out of order.

The typical way to handle this issue is to use transactional outbox pattern, i.e, store the message to be published in a table every time the data is updated. This would mean that the entire operation can be committed in a single transaction and no 2 phase commit problem is involved.

Now, this message needs to be published from the transactional outbox table to the mesaging broker for the other microservices to take corresponding actions. This can be done via one of the following approaches:

1. Polling publisher: There can be a Polling publisher which frequently checks this transactional outbox table and picks the data in the timeCreated order and publishes the messages in the same order. If the messaging broker is down or not reachable while trying to publish the message, the Polling publisher can do finite retries and quit for the next iteration to publish the message. The message publishing can get delayed slightly with this approach based on the frequency of the Polling publisher. So the frequency of the Polling publisher needs to be adjusted based on the use case. The polling publisher need to delete the data from Transactional Outbox table once the message is published.

2. Transaction Log tailing: There are various connectors available with the messaging brokers like kafka for tailing a source database and publishing the messages and even to a target sink database. Such connectors can be used for tailing the transactional outbox table and the connector can be used for publishing the messages. This approach is reliable but is database specific and a bit complex. So if the minor delay is fine for the use case, polling publisher could be the go to solution.

But Transaction Log tailing may be a requirement for a NOSQL database to handle this scenario. The NOSQL database typically will not be having a transaction and hence to store the messages in a single transaction to the database, the generated message may also be required to be stored in the same table as the one in which domain object is stored. Now the challenge will be to identify the messages that needs to be published for a polling publisher. Using transactional log tailing will help to generate messages for only the newly created/updated data eliminating this challenge. 

Connector frameworks:

Debezium

Linkedin Databus

Dyanamo DB streams

Confluent connectors

Microservices and Software Architecture Styles

  Software Architecture Styles

1. Layered Architecture - Application is classified into multiple layers like presentation layer, business layer, persistence layer in a 3 tier architecture.

2. Hexagonal Architecture - Business logic is kept independent of other layers with communications happening via inbound/outbound adapters (Interfaces).

3. Microservices Architecture - The application is divided into multiple services based on domain/maintainability etc and other factors, each service of a microservice architecture typically follows hexagonal architecture.

Advantages of microservices

1. Enables continous delivery and deployment of large complex application via independent services.

2. Independent scaling

3. Fault isolation

4. Independent technology stack and can easily update to latest tech stack

5. Independent CI/CD pipeline and faster deployment/release

6. Smaller services are be easily maintainable

7. Enables teams to be autonomous

Drawbacks of microservices architecture

1. Finding the right set of services is challenging

2. Features spanning multiple services requires careful co-ordination:

 a. how to handle transaction across services (Saga)

 b. how to query data from multiple microservices (CQRS/API Composition)

 c. may require careful rollout plan with involved microservices to not break backward compatibility etc.

3. Testing and deployment can be difficult when multiple services are involved.

SOA Vs microservices

SOA uses SOAP based webservices where as the microservices use lighter REST based or gRPC based calls.

SOA used to integrate large monolithic applications where as services are broken down as multiple micro services in a microservice architecture.

SOA based architecture have single shared database for the entire application where as microservices have its own database.

Microservices Authorization, JWT, OAuth 2.0 and SSO

 

In a microservices architecture, once the user logs in, the identity of the user needs to be communicated to all the services that handles the requests to the application and each service needs to know that this is an authenticated user and need to check whether the user have privileges for requested operation. 

The entry point to a microservice based application is an API gateway which hide the complexity due to multiple services from the end user. The gateway will contact an Authentication and Authorization server which authenticates and logs in the user and returns a JWT access token. This access token is further used in each request to verify the identity of the user from Authorization server.

JWT is a Base64 encoded string which can be decoded into a the following (all separated by "."):

1. a header with an algorithm used for signing the token, 

2. a payload which consists of basic user info like userId and iat (issued at time) and some basic info but not credentials

3. a footer consisting of a signature generated out of header and payload using the algorithm mentioned in the header and a secret key which is held only by the authorization server.

The JWT token is issued to the user based on the login using the credentials and is typically send in the further requests as a bearer token (by placing Bearer <token> in the Authorization header). The JWT token is normally saved in the cookies for the browser to send it in the further requests.

The authorization server generates a signature by extracting header and payload from the bearer token provided to it and compare with the signature in the bearer token for validating the authenticity on subsequent requests.

OAuth 2.0 

Important Terminologies:

Resource owner - The user who owns a resource (e.g. files on a google drive).

Resource server - The server which maintains the resource (e.g. google drive server)

Authorization server - The server which validates authorization to access resource server

Client application - Any client application that wants to access user resource on behalf of the user.

Grant type client credentials

In a microservice architecture, if a microservice needs to access data from another microservice:

1. The Authorization server grants a client credentials access token to the microservice

2. The authorization server validates the requests from the source microservice based on the client credential access token.

3. permission needs to be managed for the client credential access token for restricting the access of the service to APIs in the destination microservice as required. 

Grant type authorization code

If a client application wants to access certain details from a resource server on behalf of an end user:

1. The client application sends a request to authorization server requesting the access for the resource.

2. The authorization server redirects to the end user asking him to allow the client application to access the resource on his behalf, the user has to login if he has not already logged in to the authorization server.

3. Once the user gives consent for the client application to access the resource, the authorization server will send an authorization code/token to the client application.

4. The client application may store the same and calls the authorization server with the authorization code to get an access token.

5. The client application uses this access token for communicating with the resource server.

Grant type device code

OAuth 2.0 extension that enables devices with no browser or limited input capability to obtain an access token.

Grant type refresh token

This is used for getting a new access token when the original access token is expired. The original access token will have an expiry time when refresh token is required.

Handling SSO using OAuth 2.0

Although OAuth was meant for authorization, this model is extended to implement Single Sign On using grant type authorization code flow:

1. The client application sends a request to authorization server requesting the profile information of the user.

2. The authorization server redirects to the end user asking him to allow the client application to access his profile information, the user has to login if he has not already logged in to the authorization server.

3. Once the user gives consent for the client application to access his profile info, the authorization server will send an authorization code/token to the client application.

4. The client application may store the same and calls the authorization server with the authorization code to get an access token.

5. The client application uses this access token for communicating with the resource server and gets the profile info.

6. The client application trusts the authorization server and since authorization server validated the user, the user is considered valid by the client application

6. The user profile info is then set in the security context of the client app and will proceed.