Skip to main content

Leader election of Camel router through Zookeeper

Most modern distributed systems somewhere comes across the problem where only one instance of the process or job should runs and others instance of the same job or process should be standby. Apache zookeeper project provides this above functionality out of box. I am not going to describe all the usecases or solutions that could be solved by the Zookeeper. You can read the following great article for quick start.

Let me describe our use case with more details, we have apache camel component which polling some resources and only one instance of the component should run in any time. It means there should be one leader and a few followers. When master or leader goes down, any follower should replace the master. You also can use doozerd instead of zookeeper for leader election, but for me zookeeper is easy to maintain.

It's very easy to install a multi server zookeeper cluster. I have recommend you to follow the step from this blog to setup a cluster. Now when we have done our cluster setup we are ready to go for camel.
For leader election in camel we can apply ZooKeeperRoutePolicy in camel route component as follows:
ZooKeeperRoutePolicy zooKeeperRoutePolicy = new ZooKeeperRoutePolicy("zookeeper://172.18.5.36:2182/ELECTION", 1);
for details please see the link.
Once the policy is defined, it could be plugged into route as follows:
from("quartz://zookeeper/heartbeat?cron=0/50+*+*+*+*+?")
    .routePolicy(zooKeeperRoutePolicy)
    .to("jms:queue:xyz");
Above route will be execute every 50 seconds and will send message to queue xyz if it could be bind the relevant znode. Whats going on under the hood. Assume we have the znode named ELECTION in our cluster. When we have execute our first instance of this camel route, it will creates a Ephemeral znode as follows:
[zk: 172.18.5.36:2182(CONNECTED) 0] ls /ELECTION
[602-bsha-154fcb3e-2768-4d24-9803-92b9d2d9c77c0000000005]
when camel route will disconnect the znodes will be deleted automatically. If we we have started another instance of this camel route we will go following znodes:
[zk: 172.18.5.36:2182(CONNECTED) 0] ls /ELECTION
[602-bsha-61918db4-6ebe-4283-b381-c3c4688920f10000000006, 602-bsha-154fcb3e-2768-4d24-9803-92b9d2d9c77c0000000005]
By the zookeeper leader election algorithm the ephemeral znodes 602-bsha-154fcb3e-2768-4d24-9803-92b9d2d9c77c0000000005 will be the leader because it's has the smallest sequence number.
If we will stop the first camel route, 602-bsha-61918db4-6ebe-4283-b381-c3c4688920f10000000006 znode will be the leader and second camel route will start polling.
Now imagine what happens, if the first zookeeper server failed? our first camel router would be also failed because its only connected with one zookeeper server. For high availability if we want to connect to more zookeeper server from camel ZookeeperRoutePolicy we have to add more server ip address and port to the configuration. But camel ZookeeperRoutePolicy doesn't provide this functionality out of box. One of my college found a very quick fix of this problem by using ZooKeeperConfiguration class.
We have to first add the bean in spring context, it could be autowired.
<bean id="zookeeperComponent" class="org.apache.camel.component.zookeeper.ZooKeeperComponent"/>
After that we can add ZooKeeperConfiguration properties to the above component as follows:
ZooKeeperConfiguration zooConfig = new ZooKeeperConfiguration();
        zooConfig.addZookeeperServer("172.18.5.36:2181");
        zookeeperComponent.setConfiguration(zooConfig);
It's provide camel route component to bind with another zookeeper server for getting high availability. If one of the zookeeper server will fail, camel route component will still running.

Labels:

Comments

Post a Comment

Popular posts from this blog

8 things every developer should know about the Apache Ignite caching

Any technology, no matter how advanced it is, will not be able to solve your problems if you implement it improperly. Caching, precisely when it comes to the use of a distributed caching, can only accelerate your application with the proper use and configurations of it. From this point of view, Apache Ignite is no different, and there are a few steps to consider before using it in the production environment. In this article, we describe various technics that can help you to plan and adequately use of Apache Ignite as cutting-edge caching technology. Do proper capacity planning before using Ignite cluster. Do paperwork for understanding the size of the cache, number of CPUs or how many JVMs will be required. Let’s assume that you are using Hibernate as an ORM in 10 application servers and wish to use Ignite as an L2 cache. Calculate the total memory usages and the number of Ignite nodes you have to need for maintaining your SLA. An incorrect number of the Ignite nodes can become a b...
Post a Comment
Read more

Benchmarking high performance java collection framework

I am an ultimate fan of java high performance framework or library. Java native collection framework always works with primitive wrapper class such as Integer, Float e.t.c. Boxing and unboxing of wrapper class to primitive data type always decrease the java execution performance. Most of us, always looking for such a library or framework to works with primitive data type in collections for increasing performance of Java application. Most of the time i uses javolution framework to get better performance, however, this holiday i have read about a few new java collections frameworks and decided to do some homework benchmarking to find out, how much they could better than Java native collection framework. I have examine two new java collection framework, one of them are fastutil and another one are HPPC. For benchmarking i have used java JMH with mode Throughput. For benchmarking i took similar collection for java ArrayList, HashSet and HasMap from two above described frameworks. Col...
Post a Comment
Read more

Apache Ignite Baseline Topology by Examples

Ignite Baseline Topology or BLT represents a set of server nodes in the cluster that persists data on disk. Where, N1-2 and N5 server nodes are the member of the Ignite clusters with native persistence which enable data to persist on disk. N3-4 and N6 server nodes are the member of the Ignite cluster but not a part of the baseline topology. The nodes from the baseline topology are a regular server node, that store's data in memory and on the disk, and also participates in computing tasks. Ignite clusters can have different nodes that are not a part of the baseline topology such as: Server nodes that are not used Ignite native persistence to persist data on disk. Usually, they store data in memory or persists data to a 3rd party database or NoSQL. In the above equitation, node N3 or N4 might be one of them. Client nodes that are not stored shared data. To better understand the baseline topology concept, let’s start at the beginning and try to understand its goal and what ...
Post a Comment
Read more