• Articles
  • /
  • Starting with JPA and Hibernate
 

Creating a circuit breaker with Spring AOP

Intro

Most of you are probably wondering right now what a circuit breaker is. Well, it's a pattern I picked up while reading Release It! , a Pragmatic Bookshelf jewel. For those of you who haven't read this book yet, go to the bookstore and get it. It's one of my favorites and it has helped me to become a better developer.

Quoting the book, a circuit breaker 'is a component designed to fail first, thereby controlling the overall failure mode' . Put shortly, it allows you to circumvent a certain subsystem when that subsystem is not healthy.

It can be a very handy component when your system is communicating with several outside systems. When a certain system is failing over and over, you probably don't want your system to keep trying, hoping it will go up again. Using a circuit breaker, you can stop your system, failing fast and more importantly, gracefully.

Basics of the circuit breaker

When a circuitbreaker is in a closed state, all calls go through as they should. when a certain amount of failures is reached, the circuitbreaker will spring open.

In the open state, all calls are blocked and a specific failure is generated. This failure is instantaneous and can be handled gracefully.

After a certain amount of time, the circuitbreaker will go into a half-open state. The next call through the circuit breaker will attempt to do the operation, and if it succeeds, the circuitbreaker will close again, allowing subsequent calls to pass through again in a normal fashion. Should it fail, the circuit breaker will open again.

How to implement

When I first read this, I though to myself "this things screams AOP" . I envisioned annotating a method, and AOP will handle the circuit breaker aspect of that method.

Implementing this was quite easy. You just need to create the circuit breaker itself, the annotation, and create the aspect to handle its behavior.

Starting off, I created a enum to contain the states of a circuit breaker, being open, closed and half-open.

public enum CircuitBreakerStatus
{
    OPEN,
    CLOSED,
    HALF_OPEN;
}

Then I started designing the circuitbreaker. The circuit breaker had to contain its state, when the state was last changed, the failure threshold and when a retry was needed.

Additionally, I added the possibility to exclude certain exceptions to spring the circuit breaker. I.e. you want to count a IOException as a failure, but a IllegalArgumentException might not be that critical. Also, counting the times the circuit breaker was tripped might be handy for pinpointing problems within the system.

Combining all that, the circuitbreaker looks something like this:

public class CircuitBreaker
{
    private final int failureThreshold;
    private int failureCount;
    private final int timeUntilRetry;
    private CircuitBreakerStatus status;
    private long lastOpenedTime;
    private final String name;
    private int openCount;

    public CircuitBreaker(String name, int failureThreshold, int timeUntilRetry)
    {
        this.name = name;
        this.failureThreshold = failureThreshold;
        this.timeUntilRetry = timeUntilRetry;
        this.status = CircuitBreakerStatus.CLOSED;
    }

    public String getName()
    {
        return name;
    }

    public void setStatus(final CircuitBreakerStatus status)
    {
        this.status = status;
    }

    public CircuitBreakerStatus getStatus()
    {
        return status;
    }

    public void setLastOpenedTime(final long lastOpenedTime)
    {
        this.lastOpenedTime = lastOpenedTime;
    }

    public long getLastOpenedTime()
    {
        return lastOpenedTime;
    }

    public void addFailure()
    {
        failureCount++;
    }

    private void setFailureCount(final int failureCount)
    {
        this.failureCount = failureCount;
    }

    public int getFailureCount()
    {
        return failureCount;
    }

    public boolean isWaitTimeExceeded()
    {
        return System.currentTimeMillis() - timeUntilRetry > lastOpenedTime;
    }

    public boolean isThresholdReached()
    {
        return getFailureCount() >= getFailureThreshold();
    }

    public int getFailureThreshold()
    {
        return failureThreshold;
    }

    public void open()
    {
        setLastOpenedTime(System.currentTimeMillis());
        setStatus(CircuitBreakerStatus.OPEN);
        openCount++;
    }

    public void openHalf()
    {
        setStatus(CircuitBreakerStatus.HALF_OPEN);
    }

    public void close()
    {
        setFailureCount(0);
        setStatus(CircuitBreakerStatus.CLOSED);
    }

    public boolean isOpen()
    {
        return getStatus() == CircuitBreakerStatus.OPEN;
    }

    public boolean isHalfOpen()
    {
        return getStatus() == CircuitBreakerStatus.HALF_OPEN;
    }

    public boolean isClosed()
    {
        return getStatus() == CircuitBreakerStatus.CLOSED;
    }

    public int getOpenCount()
    {
        return openCount;
    }
}

Off course, we're missing the engine for the circuit breaker, the AOP method interceptor. In short, it handles the method calls and if a method is annotated with @UseCircuitBreaker , it will add circuitbreaker functionality to that method.

The code looks something like this:

public class CircuitBreakerInterceptor implements MethodInterceptor
{
    private Map<String, CircuitBreaker> breakers = new HashMap<String, CircuitBreaker>();
    private transient final Log log = LogFactory.getLog(getClass());
    private List<Class<? extends Throwable>> nonTrippingExceptions;
    private final int failureThreshold;
    private final int timeUntilRetry;

    public CircuitBreakerInterceptor(int failureThreshold, int timeUntilRetry)
    {
        this.failureThreshold = failureThreshold;
        this.timeUntilRetry = timeUntilRetry;
    }

    public CircuitBreaker getBreaker(String name)
    {
        if (breakers.get(name) == null)
        {
            breakers.put(name, new CircuitBreaker(name, failureThreshold, timeUntilRetry));
        }
        return breakers.get(name);
    }

    private boolean isNonTrippingException
            (Throwable
                    t)
    {
        for (Class throwable : nonTrippingExceptions)
        {
            if (throwable.isAssignableFrom(t.getClass()))
            {
                return true;
            }
        }
        return false;
    }

    public List<Class<? extends Throwable>> getNonTrippingExceptions
            ()
    {
        if (nonTrippingExceptions == null)
            nonTrippingExceptions = new ArrayList<Class<? extends Throwable>>();
        return nonTrippingExceptions;
    }

    public void setNonTrippingExceptions
            (
                    final List<Class<? extends Throwable>> nonTrippingExceptions)
    {
        this.nonTrippingExceptions = nonTrippingExceptions;
    }

    public Object invoke(final MethodInvocation invocation) throws Throwable
    {
        log.info("Going through circuitbreaker protected method");
        UseCircuitBreaker breakerAnnot = invocation.getMethod().getAnnotation(UseCircuitBreaker.class);
        if (breakerAnnot != null)
        {
            CircuitBreaker breaker = getBreaker(breakerAnnot.value());
            Object returnValue = null;
            if (breaker.isOpen())
            {
                if (breaker.isWaitTimeExceeded())
                {
                    try
                    {
                        breaker.openHalf();
                        log.info("Retrying operation " + invocation.getMethod().toGenericString() + " after waitTime exceeded");
                        returnValue = invocation.proceed();
                        log.info("Retry of operation " + invocation.getMethod().toGenericString() + " succeeded, resetting circuit breaker");
                        breaker.close();
                    }
                    catch (Throwable t)
                    {
                        if (isNonTrippingException(t))
                        {
                            log.info("Retry of operation " + invocation.getMethod().toGenericString() + " succeeded (but threw legal exception), resetting circuit breaker", t);
                            breaker.close();
                            throw t;
                        }
                        log.error("Retry of operation " + invocation.getMethod().toGenericString() + " failed, circuit breaker still closed.", t);
                        breaker.open();
                        throw new CircuitBreakerException("The operation " + invocation.getMethod().toGenericString() + " has too many failures, tripping circuit breaker.", t);
                    }
                }
                else
                {
                    throw new CircuitBreakerException("This operation cannot be performed due to open circuit breaker (too many failures).");
                }
            }
            else if (breaker.isClosed())
            {
                try
                {
                    returnValue = invocation.proceed();
                    breaker.close();
                }
                catch (Throwable t)
                {
                    log.info("Failure of operation " + invocation.getMethod().toGenericString() + " in circuit breaker", t);
                    if (!isNonTrippingException(t))
                    {
                        breaker.addFailure();
                        if (breaker.isThresholdReached())
                        {
                            log.error("Circuit breaker tripped on operation " + invocation.getMethod().toGenericString() + " failure.", t);
                            breaker.open();
                            throw new CircuitBreakerException("The operation " + invocation.getMethod().toGenericString() + " has too many failures, tripping circuit breaker.", t);
                        }
                        else
                        {
                            throw t;
                        }
                    }
                    else
                    {
                        throw t;
                    }
                }
            }
            else if (breaker.isHalfOpen())
            {
                throw new CircuitBreakerException("Busy retrying operation in opened circuit breaker");
            }
            return returnValue;
        }
        else
        {
            return invocation.proceed();
        }
    }
}

Using the circuit breaker interceptor in your Spring code

Using the interceptor is very easy and can be done in the way described in the Spring manual.

I've created a service to demonstrate the features of the circuit breaker. It consists of am interface and a demo implementation of that interface.

public interface ICircuitBreakerService
{
    public static final String BREAKER_ID = "MY_BREAKER";

    @UseCircuitBreaker(BREAKER_ID)
    public void withoutProblem();

    @UseCircuitBreaker(BREAKER_ID)
    public void withProblem();

    @UseCircuitBreaker(BREAKER_ID)
    public void withIgnoredProblem();
}
public class CircuitBreakerService implements ICircuitBreakerService
{
    private final transient Log log = LogFactory.getLog(getClass());

    public void withoutProblem()
    {
        log.info("without problems");
    }

    public void withProblem()
    {
         throw new BreakingException("breaking exception!");
    }

    public void withIgnoredProblem()
    {
         throw new IllegalArgumentException("illegal argument");
    }

    public static class BreakingException extends RuntimeException
    {
        public BreakingException(final String s)
        {
            super(s);
        }
    }
}

We're going to configure the circuit breaker so that it'll open when 3 BreakingException s occur, ignoring IllegalArgumentException s. The circuit breaker is allowed to retry the real operation if it has been open for more than 2 seconds (in real life, this will be a bit longer, more like 5 minutes).

Looking at the Spring configuration, you're going to have something like this:

<?xml version="1.0"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">

  <bean id="circuitBreakerService" class="be.doclo.releaseit.circuitbreaker.CircuitBreakerService"/>

  <bean id="circuitBreakerInterceptor" class="be.doclo.releaseit.circuitbreaker.CircuitBreakerInterceptor">
    <constructor-arg index="0" value="3"/>
    <constructor-arg index="1" value="2000"/>
    <property name="nonTrippingExceptions">
      <list>
        <value>java.lang.IllegalArgumentException</value>
      </list>
    </property>
  </bean>

  <bean class="org.springframework.aop.framework.autoproxy.BeanNameAutoProxyCreator">
    <property name="beanNames" value="*Service"/>
    <property name="interceptorNames">
      <list>
        <value>circuitBreakerInterceptor</value>
      </list>
    </property>
  </bean>

</beans>

When calling the withProblem() method, the method will fail 2 times with the BreakingException , but the 3rd time the threshold has been reached and the circuit breaker will open, thowing a CircuitBreakerException instead when calling that method again. This exception can then be handled gracefully.

Possible improvements

The circuit breakers are thread-safe yet, so using my code in multi-threaded environments will probably fail. Anyone who feels up to the challenge of making it thread-safe, knock yourself out (and let me know please).

Conclusion

A circuit breaker can be a really useful component when the system is connecting to other systems, and you want to control graceful degregation when one of those systems goes down.

In the book, circuit breakers are linked to time-outs. When a method takes too long to complete too many times, it might be a sign that something is wrong. Or quoting the book 'When it hurts, don't do it' .

 
2009 © Lieven Doclo