Resilient, are you ready for your application?

UP1: If you are planning to use Reactive programming, we recommend you to read the book "High performance in-memory computing with Apache Ignite".

Nowadays, term Reactive programming and Reactive manifesto is became trend. Blog post, articles and presentations are all over. Why peoples getting crazy with it? or what's the problem it could solve? The easiest way to answer the above questions is to think about the requirements we have building application these days.
We have to:
1) Modular - Module can help to maintain the application, it can be go offline and getting back to online without breaking all the system down;
2) Scalable - this way we can scale vertically or horizontally to handle a large amount of data or transactions;
3) Resilient - system can be getting back online and be fault tolerant;
4) Responsive - this means fast and available.
Here is the paradigm of reactive manifesto:

For most of the part, we already have a few framework such as Akka, Vert.x, Hystrix, ReactiveFx to develop reactive application. Today i am going to highlight only on resilient properties of any application. Probably in your development process, you had to had used any third party libraries or any 3rd party services through network. When these type of 3rd party services or libraries going down or not available, your application suffers in timeout exception or even more crash for high value waiting requests. For example, in our last project we are using 3rd party weather forecast service and define residence of city through ip address in our portal. When these services not available or not responses in time, response time of the portal increases and a few times system crashes because a lot of requests waited in thread pool. These could be happen in any external resources such as database connection, MQ connection e.t.c. If you have to recover from these situation, we have to need short circuit design pattern. Company Netflix a few years ago developed a framework to design such application, which also contain bulkhead patterns. Today i am going to describe the framework Hystrix and will develop a very quick start maven project to show it's capabilities.
Here is the illustration, how it works:

Design is very simple, "When you use Hystrix to wrap each underlying dependency. Each dependency is isolated from one other, restricted in the resources it can saturate when latency occurs, and covered in fallback logic that decides what response to make when any type of failure occurs in the dependency".
To show it's capabilities we developed a simple rest service with two methods (getweather, getcitiybyip), which will play as a 3rd party services.

public interface IServices {
    public String getWeather(String cityName);
    public String getCityByIp(String ipAddress);
}

And we also have a service facade of rest services based on Hystrix, which will delegate the request to the 3rd party services. If the 3rd party service not available, the circuit will open and no request will delegate to the service.
Here is the source code of the Hystrix service facade:

package com.blu.rest;

import com.netflix.hystrix.HystrixCommand;
import com.netflix.hystrix.HystrixCommandGroupKey;
import com.netflix.hystrix.HystrixCommandProperties;
import com.netflix.hystrix.HystrixThreadPoolProperties;
import com.sun.jersey.api.client.Client;
import com.sun.jersey.api.client.ClientResponse;
import com.sun.jersey.api.client.WebResource;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.core.Response;

/**
 * Created by shamim on 16/07/15.
 * Service facade
 */
// add json
@Path("/servicefacade")
public class HystrixServiceFacade extends HystrixCommand{
    private static final Logger LOGGER = LoggerFactory.getLogger(HystrixServiceFacade.class);
    private static final String CONTEXT_ROOT="http://localhost:8080/test/rest/serviceimpl/";
    private String ipAddress;


    public HystrixServiceFacade() {
        super(Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("getCityByIp"))
                        .andThreadPoolPropertiesDefaults(HystrixThreadPoolProperties.Setter().withCoreSize(100))
                        .andCommandPropertiesDefaults(HystrixCommandProperties.Setter().withCircuitBreakerErrorThresholdPercentage(3)
                        .withCircuitBreakerSleepWindowInMilliseconds(2000)
                        )
        );
    }

    @GET
    @Path("/getcitybyip/{param}")
    // http://localhost:8080/test/rest/servicefacade/getcitybyip/192.168.121.11
    public Response echo (@PathParam("param") String ipAddress){
        LOGGER.info("Invoked with Parameter!!!" + ipAddress);
        this.ipAddress = ipAddress;


        String resp = this.execute();
        return Response.status(200).entity(resp).build();
    }

    @Override
    protected String run() throws Exception {
        // invoke 3rd party service
        final String uri = CONTEXT_ROOT + "getcitybyip/" +this.ipAddress;
        Client client = Client.create();
        WebResource webResource = client.resource(uri);
        ClientResponse response = webResource.accept("application/json").get(ClientResponse.class);

        if (response.getStatus() != 200) {
            LOGGER.info("Error:" + response.getStatus());
        }
        return response.getEntity(String.class);


    }

    @Override
    protected String getFallback() {
        return "Fall back for get City by Ip";
    }
    private GetWeatherCommand getWeatherCommand = new GetWeatherCommand();

    @GET
    @Path("/getweather/{param}")
    public Response getWeather(@PathParam("param")String cityName){
        LOGGER.info("Invoked getWeather with param: "+ cityName);
        getWeatherCommand.cityName = cityName;
        String resp = getWeatherCommand.execute();

        return Response.status(200).entity(resp).build();
    }
    class GetWeatherCommand extends HystrixCommand{
        private String cityName;
        public GetWeatherCommand() {
            super(Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("GetWeather"))
                            .andThreadPoolPropertiesDefaults(HystrixThreadPoolProperties.Setter().withCoreSize(100))
                            .andCommandPropertiesDefaults(HystrixCommandProperties.Setter().withCircuitBreakerErrorThresholdPercentage(5)
                            .withCircuitBreakerSleepWindowInMilliseconds(2000)
                            )
            );
        }

        @Override
        protected String run() throws Exception {
            // invoke 3rd party service
            final String uri = CONTEXT_ROOT +"getweather/"+this.cityName;
            Client client = Client.create();
            WebResource webResource = client.resource(uri);
            ClientResponse response = webResource.accept("application/json").get(ClientResponse.class);

            if (response.getStatus() != 200) {
                LOGGER.info("Error {}", response.getStatus());
            }
            return response.getEntity(String.class);
        }
        // static fall back
        @Override
        protected String getFallback() {
            return "Fall Back for getWeather";
        }
    };
}

Note that, here is two type of implementation, one when we inherited from Hystrix command, another one when we are using inner Hystrix command. Full source code you will found here in github.com.
Now lets run the simple rest service by the following command:

mvn jetty:run

if everything goes well, your service should be available in url http://localhost:8080/test/rest/serviceimpl/getweather/moscow
Now lets run the hystrix service facade service

mvn jetty:run -Djetty.http.port=9093

Service facade will be available in port 9093 and you can reach it by http://localhost:9093/test/rest/servicefacade/getweather/moscow
Also i have configured HystrixStreamServlet for sending matrices to Hystrix Dashboard.

<servlet>
    <description></description>
    <display-name>HystrixMetricsStreamServlet</display-name>
    <servlet-name>HystrixMetricsStreamServlet</servlet-name>
    <servlet-class>com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet</servlet-class>
</servlet>

<servlet-mapping>
    <servlet-name>HystrixMetricsStreamServlet</servlet-name>
    <url-pattern>/hystrix.stream</url-pattern>
</servlet-mapping>

StreamServlet will available in http://localhost:9093/test/hystrix.stream
Also you can clone the Hystrix dash board from github and follow the instruction to build and run the dashboard. In my case dashboard is available in http://localhost:7979/hystrix-dashboard.
Now we can start sending massive requests to check how it works. For these you can use any Curl command or using my com.blu.reactive.JersyClient class to start sending request. For demonstration purpose of network failure i have add following randomly failure code to the getCityByIp method as follows:

package com.blu.rest;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import java.util.HashMap;
import java.util.Map;

@Path("/serviceimpl")
public class ServicesImpl implements IServices {
    private static final Map WEATHERS= new HashMap(){{
        put("moscow", "29");
        put("petersburg", "21");
        put("berlin", "31");
        put("bercelona", "33");
    }};
    private static final Map CITIES = new HashMap(){{
        put("192.168.121.11","moscow");
        put("192.168.124.21","petersburg");
        put("192.138.111.31","berlin");
        put("192.168.132.61","bercelona");
    }};

    @GET
    @Path("/getweather/{param}")
    @Override
    public String getWeather(@PathParam("param") String cityName) {
        return WEATHERS.get(cityName);
    }
    @GET
    @Path("/getcitybyip/{param}")
    @Override
    public String getCityByIp(@PathParam("param")String ipAddress) {
        // Randomly failure
        final long ctime = System.currentTimeMillis();

        if(ctime % 20 == 0){
            throw new RuntimeException("Randomly failure!!");
        }
        if(ctime % 30 == 0){
            try {
                Thread.sleep(7000);
            } catch (InterruptedException e) {
                // do nothing
            }
        }
        return CITIES.get(ipAddress);
    }
}

After running the JersyClient class, you can observed the Hystrix Dashboard and monitor the services health.
When there is no service failure, circuit is closed and delegated to the 3rd party service as follows:

When randomly failure occurs or you stop the 3rd party service, Hystrix discovered the network failure and circuit is open, then no service requests delegates to the 3rd party services and hystrix service facade returns the fallback. After 2 second (it configurable by withCircuitBreakerSleepWindowInMilliseconds(2000)) hystrix passes a few requests and if the service keep alive it continue to passed all the request. Here is the illustration when circuit is open:

Certainly you can control how much service failure will cause circuit open, in our case it's 5% of all request. In Hystrix framework there are a few many interesting capabilities such as bulkhead, timeout control, but for today it's enough. Happy Weekend.