Skip to main content

Resilient, are you ready for your application?

UP1: If you are planning to use Reactive programming, we recommend you to read the book "High performance in-memory computing with Apache Ignite".

Nowadays, term Reactive programming and Reactive manifesto is became trend. Blog post, articles and presentations are all over. Why peoples getting crazy with it? or what's the problem it could solve? The easiest way to answer the above questions is to think about the requirements we have building application these days.
We have to:
1) Modular - Module can help to maintain the application, it can be go offline and getting back to online without breaking all the system down;
2) Scalable - this way we can scale vertically or horizontally to handle a large amount of data or transactions;
3) Resilient - system can be getting back online and be fault tolerant;
4) Responsive - this means fast and available.
Here is the paradigm of reactive manifesto:
For most of the part, we already have a few framework such as Akka, Vert.x, Hystrix, ReactiveFx to develop reactive application. Today i am going to highlight only on resilient properties of any application. Probably in your development process, you had to had used any third party libraries or any 3rd party services through network. When these type of 3rd party services or libraries going down or not available, your application suffers in timeout exception or even more crash for high value waiting requests. For example, in our last project we are using 3rd party weather forecast service and define residence of city through ip address in our portal. When these services not available or not responses in time, response time of the portal increases and a few times system crashes because a lot of requests waited in thread pool. These could be happen in any external resources such as database connection, MQ connection e.t.c. If you have to recover from these situation, we have to need short circuit design pattern. Company Netflix a few years ago developed a framework to design such application, which also contain bulkhead patterns. Today i am going to describe the framework Hystrix and will develop a very quick start maven project to show it's capabilities.
Here is the illustration, how it works:
Design is very simple, "When you use Hystrix to wrap each underlying dependency. Each dependency is isolated from one other, restricted in the resources it can saturate when latency occurs, and covered in fallback logic that decides what response to make when any type of failure occurs in the dependency".
To show it's capabilities we developed a simple rest service with two methods (getweather, getcitiybyip), which will play as a 3rd party services.
public interface IServices {
    public String getWeather(String cityName);
    public String getCityByIp(String ipAddress);
}
And we also have a service facade of rest services based on Hystrix, which will delegate the request to the 3rd party services. If the 3rd party service not available, the circuit will open and no request will delegate to the service.
Here is the source code of the Hystrix service facade:
package com.blu.rest;

import com.netflix.hystrix.HystrixCommand;
import com.netflix.hystrix.HystrixCommandGroupKey;
import com.netflix.hystrix.HystrixCommandProperties;
import com.netflix.hystrix.HystrixThreadPoolProperties;
import com.sun.jersey.api.client.Client;
import com.sun.jersey.api.client.ClientResponse;
import com.sun.jersey.api.client.WebResource;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.core.Response;

/**
 * Created by shamim on 16/07/15.
 * Service facade
 */
// add json
@Path("/servicefacade")
public class HystrixServiceFacade extends HystrixCommand{
    private static final Logger LOGGER = LoggerFactory.getLogger(HystrixServiceFacade.class);
    private static final String CONTEXT_ROOT="http://localhost:8080/test/rest/serviceimpl/";
    private String ipAddress;


    public HystrixServiceFacade() {
        super(Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("getCityByIp"))
                        .andThreadPoolPropertiesDefaults(HystrixThreadPoolProperties.Setter().withCoreSize(100))
                        .andCommandPropertiesDefaults(HystrixCommandProperties.Setter().withCircuitBreakerErrorThresholdPercentage(3)
                        .withCircuitBreakerSleepWindowInMilliseconds(2000)
                        )
        );
    }

    @GET
    @Path("/getcitybyip/{param}")
    // http://localhost:8080/test/rest/servicefacade/getcitybyip/192.168.121.11
    public Response echo (@PathParam("param") String ipAddress){
        LOGGER.info("Invoked with Parameter!!!" + ipAddress);
        this.ipAddress = ipAddress;


        String resp = this.execute();
        return Response.status(200).entity(resp).build();
    }

    @Override
    protected String run() throws Exception {
        // invoke 3rd party service
        final String uri = CONTEXT_ROOT + "getcitybyip/" +this.ipAddress;
        Client client = Client.create();
        WebResource webResource = client.resource(uri);
        ClientResponse response = webResource.accept("application/json").get(ClientResponse.class);

        if (response.getStatus() != 200) {
            LOGGER.info("Error:" + response.getStatus());
        }
        return response.getEntity(String.class);


    }

    @Override
    protected String getFallback() {
        return "Fall back for get City by Ip";
    }
    private GetWeatherCommand getWeatherCommand = new GetWeatherCommand();

    @GET
    @Path("/getweather/{param}")
    public Response getWeather(@PathParam("param")String cityName){
        LOGGER.info("Invoked getWeather with param: "+ cityName);
        getWeatherCommand.cityName = cityName;
        String resp = getWeatherCommand.execute();

        return Response.status(200).entity(resp).build();
    }
    class GetWeatherCommand extends HystrixCommand{
        private String cityName;
        public GetWeatherCommand() {
            super(Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("GetWeather"))
                            .andThreadPoolPropertiesDefaults(HystrixThreadPoolProperties.Setter().withCoreSize(100))
                            .andCommandPropertiesDefaults(HystrixCommandProperties.Setter().withCircuitBreakerErrorThresholdPercentage(5)
                            .withCircuitBreakerSleepWindowInMilliseconds(2000)
                            )
            );
        }

        @Override
        protected String run() throws Exception {
            // invoke 3rd party service
            final String uri = CONTEXT_ROOT +"getweather/"+this.cityName;
            Client client = Client.create();
            WebResource webResource = client.resource(uri);
            ClientResponse response = webResource.accept("application/json").get(ClientResponse.class);

            if (response.getStatus() != 200) {
                LOGGER.info("Error {}", response.getStatus());
            }
            return response.getEntity(String.class);
        }
        // static fall back
        @Override
        protected String getFallback() {
            return "Fall Back for getWeather";
        }
    };
}
Note that, here is two type of implementation, one when we inherited from Hystrix command, another one when we are using inner Hystrix command. Full source code you will found here in github.com.
Now lets run the simple rest service by the following command:
mvn jetty:run
if everything goes well, your service should be available in url http://localhost:8080/test/rest/serviceimpl/getweather/moscow
Now lets run the hystrix service facade service
mvn jetty:run -Djetty.http.port=9093
Service facade will be available in port 9093 and you can reach it by http://localhost:9093/test/rest/servicefacade/getweather/moscow
Also i have configured HystrixStreamServlet for sending matrices to Hystrix Dashboard.
<servlet>
    <description></description>
    <display-name>HystrixMetricsStreamServlet</display-name>
    <servlet-name>HystrixMetricsStreamServlet</servlet-name>
    <servlet-class>com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet</servlet-class>
</servlet>

<servlet-mapping>
    <servlet-name>HystrixMetricsStreamServlet</servlet-name>
    <url-pattern>/hystrix.stream</url-pattern>
</servlet-mapping>
StreamServlet will available in http://localhost:9093/test/hystrix.stream
Also you can clone the Hystrix dash board from github and follow the instruction to build and run the dashboard. In my case dashboard is available in http://localhost:7979/hystrix-dashboard.
Now we can start sending massive requests to check how it works. For these you can use any Curl command or using my com.blu.reactive.JersyClient class to start sending request. For demonstration purpose of network failure i have add following randomly failure code to the getCityByIp method as follows:
package com.blu.rest;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import java.util.HashMap;
import java.util.Map;

@Path("/serviceimpl")
public class ServicesImpl implements IServices {
    private static final Map WEATHERS= new HashMap(){{
        put("moscow", "29");
        put("petersburg", "21");
        put("berlin", "31");
        put("bercelona", "33");
    }};
    private static final Map CITIES = new HashMap(){{
        put("192.168.121.11","moscow");
        put("192.168.124.21","petersburg");
        put("192.138.111.31","berlin");
        put("192.168.132.61","bercelona");
    }};

    @GET
    @Path("/getweather/{param}")
    @Override
    public String getWeather(@PathParam("param") String cityName) {
        return WEATHERS.get(cityName);
    }
    @GET
    @Path("/getcitybyip/{param}")
    @Override
    public String getCityByIp(@PathParam("param")String ipAddress) {
        // Randomly failure
        final long ctime = System.currentTimeMillis();

        if(ctime % 20 == 0){
            throw new RuntimeException("Randomly failure!!");
        }
        if(ctime % 30 == 0){
            try {
                Thread.sleep(7000);
            } catch (InterruptedException e) {
                // do nothing
            }
        }
        return CITIES.get(ipAddress);
    }
}
After running the JersyClient class, you can observed the Hystrix Dashboard and monitor the services health.
When there is no service failure, circuit is closed and delegated to the 3rd party service as follows:
When randomly failure occurs or you stop the 3rd party service, Hystrix discovered the network failure and circuit is open, then no service requests delegates to the 3rd party services and hystrix service facade returns the fallback. After 2 second (it configurable by withCircuitBreakerSleepWindowInMilliseconds(2000)) hystrix passes a few requests and if the service keep alive it continue to passed all the request. Here is the illustration when circuit is open:

Certainly you can control how much service failure will cause circuit open, in our case it's 5% of all request. In Hystrix framework there are a few many interesting capabilities such as bulkhead, timeout control, but for today it's enough. Happy Weekend.

Comments

Popular posts from this blog

Send e-mail with attachment through OSB

Oracle Service Bus (OSB) contains a good collection of adapter to integrate with any legacy application, including ftp, email, MQ, tuxedo. However e-mail still recognize as a stable protocol to integrate with any application asynchronously. Send e-mail with attachment is a common task of any business process. Inbound e-mail adapter which, integrated with OSB support attachment but outbound adapter doesn't. This post is all about sending attachment though JavaCallout action. There are two ways to handle attachment in OSB: 1) Use JavaCallout action to pass the binary data for further manipulation. It means write down a small java library which will get the attachment and send the e-mail. 2) Use integrated outbound e-mail adapter to send attachment, here you have to add a custom variable named attachment and assign the binary data to the body of the attachment variable. First option is very common and easy to implement through javax.mail api, however a much more developer manage t

Tip: SQL client for Apache Ignite cache

A new SQL client configuration described in  The Apache Ignite book . If it got you interested, check out the rest of the book for more helpful information. Apache Ignite provides SQL queries execution on the caches, SQL syntax is an ANSI-99 compliant. Therefore, you can execute SQL queries against any caches from any SQL client which supports JDBC thin client. This section is for those, who feels comfortable with SQL rather than execute a bunch of code to retrieve data from the cache. Apache Ignite out of the box shipped with JDBC driver that allows you to connect to Ignite caches and retrieve distributed data from the cache using standard SQL queries. Rest of the section of this chapter will describe how to connect SQL IDE (Integrated Development Environment) to Ignite cache and executes some SQL queries to play with the data. SQL IDE or SQL editor can simplify the development process and allow you to get productive much quicker. Most database vendors have their own front-en

Load balancing and fail over with scheduler

Every programmer at least develop one Scheduler or Job in their life time of programming. Nowadays writing or developing scheduler to get you job done is very simple, but when you are thinking about high availability or load balancing your scheduler or job it getting some tricky. Even more when you have a few instance of your scheduler but only one can be run at a time also need some tricks to done. A long time ago i used some data base table lock to achieved such a functionality as leader election. Around 2010 when Zookeeper comes into play, i always preferred to use Zookeeper to bring high availability and scalability. For using Zookeeper you have to need Zookeeper cluster with minimum 3 nodes and maintain the cluster. Our new customer denied to use such a open source product in their environment and i was definitely need to find something alternative. Definitely Quartz was the next choose. Quartz makes developing scheduler easy and simple. Quartz clustering feature brings the HA and