Circuit Breakers and Microservices Architecture

By now it’s pretty well-known that a Microservices architecture has many advantages. These include low coupling, re-usability, business agility and distributed cloud ready applications. But at the same time it makes the architecture brittle because each user action results invokes multiple services. It replaces the in-memory calls of a monolithic architecture with remote calls across the network, which works well when all services are up and running. But when one or more services are unavailable or exhibiting high latency, it results in a cascading failure across the enterprise. Service client retry logic only makes things worse for the service, and can bring it down completely.

You can read more on Microservices architectures at Constant Contact.

The Circuit breaker pattern helps to prevent such a catastrophic cascading failure across multiple systems. The circuit breaker pattern allows you to build a fault tolerant and resilient system that can survive gracefully when  key services are either unavailable or have high latency.

Everything fails, accept it!

Let’s face it, all services will fail or falter at some point in time. Circuit breakers allow your system to handle these failures gracefully. The circuit breaker concept is straightforward. It wraps a function with a monitor that tracks failures. The circuit breaker has 3 distinct states, Closed, Open, and Half-Open:

  • Closed – When everything is normal, the circuit breaker remains in the closed state and all calls pass through to the services. When the number of failures exceeds a predetermined threshold the breaker trips, and it goes into the Open state.
  • Open – The circuit breaker returns an error for calls without executing the function.
  • Half-Open – After a timeout period, the circuit switches to a half-open state to test if the underlying problem still exists. If a single call fails in this half-open state, the breaker is once again tripped. If it succeeds, the circuit breaker resets back to the normal closed state. 

Here’s an explanation and flow diagram of each circuit-breaker state.

Circuit Breakers in CLOSED State

When the circuit breaker is in the CLOSED state, all calls go through to the Supplier Microservice, which  responds without any latency.

Circut Breaker Closed State - When the circuit breaker is in the CLOSED state, all calls go through to the Supplier Microservice, which responds without any latency.

Circuit Breakers in OPEN State

If the Supplier Microservice is experiencing slowness, the circuit breaker receives timeouts for any requests to that service. Once number of timeouts reaches a predetermined threshold,  it trips the circuit breaker to the OPEN state. In the OPEN state the circuit breaker returns an  error for all calls to the service without making the calls to the Supplier Microservice. This behavior allows the Supplier Microservice to recover  by reducing its load.

If the Supplier Microservice is experiencing slowness, the circuit breaker receives timeouts for its requests to the it. Once number of timeouts reaches the predetermined threshold, it trips the circuit breaker to the OPEN state. In theOPEN state, all calls are returned with an error by the circuit breaker, without making calls to the Supplier Microservice. This behavior helps the Supplier Microservice recover by reducing its load.

Circuit Breakers in HALF-OPEN State

The circuit breaker uses a monitoring and feedback mechanism called the HALF-OPEN state to know if and when the Supplier Microservice has recovered. It uses this mechanism to make a trial call to the supplier microservice periodically to check if it has recovered. If the call to the Supplier Microservice times out, the circuit breaker remains in the OPEN state. If the call returns success, then the circuit switches to the CLOSED state. The circuit breaker then returns all external calls to the service with an error during the HALF-OPEN state. 

In order for the circuit breaker to know if the Supplier Microservice has recovered, it needs a monitoring and feedback mechanism. It uses this mechanism to make a trial call to the supplier microservice periodically to check if it has recovered. This is called the HALF-OPEN state. If the call to the supplier microservice times out, the circuit breaker remains in the OPEN state. If the call returns success, then the circuit will be switched to CLOSED state.

Example Circuit Breaker using Spring boot

If you use Spring Boot, you can use the circuit breaker implementation from Netflix Hystrix fault tolerance library.

@EnableCircuitBreaker
@RestController
@SpringBootApplication
public class CampaignApplication {

  @Autowired
  private TrackingService trackingService;

  @Bean
  public RestTemplate rest(RestTemplateBuilder builder) {
    return builder.build();
  }

  @RequestMapping("/to-track")
  public String toTrack() {
    return trackingService.getStats();
  }

  public static void main(String[] args) {
    SpringApplication.run(ReadingApplication.class, args);
  }
}

The “EnableCircuitBreaker” annotation tells Spring Cloud that the Campaign application uses circuit breakers that will enable their monitoring, opening, and closing as per availability of tracking service.

Advantages

The circuit breaker is valuable for monitoring; monitor, log, and recover any circuit breaker state changes to ensure service availability. Testing circuit breaker states helps you to add logic for a fault tolerant system. For example, if the product catalog service is not available (the circuit is open), then the UI can fetch the product list from the cache. The circuit breaker pattern handles downtime and slowness of key services gracefully and helps those services recover by reducing load.

Additional Reading

Leave a Comment