My workspace

One of the speaker in last JavaOne Moscow mentioned about JVM multi-tenancy and declare it's availability in Java 9. End of the last year IBM also declare their IBM J9 JVM to host multiple applications. Out of curiosity i was interested on this topics and time to time read a few articles and see presentation. Who hasn't familiar with JVM multi-tenancy yet, JVM multi-tenancy is hosting more then one application on single JVM. Here you will found a very good articles about JVM multi-tanancy. Last Friday i have got an e-mail from Dzone mailing list about ElastiCat , which providing jvm multi tenancy and much more. After spending some time i download the products and deploy simple web service on it and play with JVM. Today i have found a few spare hours after lunch and decided to share my experience with ElastiCat. ElastiCat, as they declared, provides Multitenancy, Isolation, Density and Elasticity through Tomcat. You can deploy more than one application on single tomcat and sha...

One of the main disadvantage of using PIG is that, Pig always raise all the data from Cassandra Storage, and after that it can filter by your choose. It's very easy to imagine how the workload will be if you have a tons of million rows in your CF. For example, in our production environment we have always more than 300 million rows, where only 20-25 millions of rows is unprocessed. When we are executing pig script, we have got more than 5000 map tasks with all the 300 millions of rows. It's time consuming and high load batch processing we always tried to avoid but in vain. It's could be very nice if we could use CQL query in pig scripts with where clause to select and filter our data. Here benefit is clear, less data will consume, less map task and a little workload. Still in latest version of Cassandra (1.2.6) this feature is not available. This feature is planned in next version Cassandra 1.2.7. However patch is already available for this feature, with a few efforts we ...

Most modern distributed systems somewhere comes across the problem where only one instance of the process or job should runs and others instance of the same job or process should be standby. Apache zookeeper project provides this above functionality out of box. I am not going to describe all the usecases or solutions that could be solved by the Zookeeper. You can read the following great article for quick start. Let me describe our use case with more details, we have apache camel component which polling some resources and only one instance of the component should run in any time. It means there should be one leader and a few followers. When master or leader goes down, any follower should replace the master. You also can use doozerd instead of zookeeper for leader election, but for me zookeeper is easy to maintain. It's very easy to install a multi server zookeeper cluster. I have recommend you to follow the step from this blog to setup a cluster. Now when we have done our cl...

After a few weeks break at last we completed our tuning and configuration cassandra hadoop stack in production. It was exciting and i decided to share our experience with all. 1) Cassandra version >> 1.2 has some problems and doesn't integrate with Hadoop very well. The problem with Map Reduce, when we runs any Map reduce job, it always assigns only one mapper regardless of the amount of data. See here for more detail. 2) If you are going to use Pig for you data analysis, think twice, because Pig always picks up all the data from the Cassandra Storage and only after these it can filter. If you have a billions of rows and only a few millions of then you have to aggregate, then Pig always pick up the billions of rows. Here you can find a compression between Hadoop framework for executing Map reduce. 3) If you are using Pig, filter rows as early as possible. Filter fields like null or empty. 4) When using Pig, try to model your CF slightly different. Use Bucket pattern, sto...

Last couple of months we are using Apache Hadoop Map Reduce batch processing to analyze a huge amount of data. We have a few legacy product where we can't consider to using Cassandra big table database. A few of them uses Oracle Database as their primary storage. As our requirements we have to extract the data from the rdbms, parse the payload and load it to Cassandra for aggregating. Here i have decided to use Storm for real time data processing. Our usecase is as follows: 1) Storm spout connects to Oracle Database and collects data from particular table with some intervals. 2) Storm bolt parses the data with some fashion and emit to Storm-cassandra bolt to store the row into Cassandra DB. Here is the fragment code of project. First i have create a Jdbc connector class, class contain a few class variables which contradicting with Storm ideology, as far i have just need one spout as input - it's enough for me. package storm.contrib.jdbc; import org.slf4j.Logger; import ...

Today our team tried to install JNA 3.5.0 in our UAT environment. Here is the link to make a try. At the moment of Cassandra start we have noticed the following INFO on cassandra log: INFO [main] 2012-11-27 13:20:17,747 CLibrary.java (line 66) JNA link failure, one or more native method will be unavailable. Very interesting thing is that, most of the JNA features works. I have decided to investigate the problem and restart Cassandra in debug mode (edit the log4j-server.properties and set the rootLogger level DEBUG) and found the details error DEBUG [main] 2012-11-27 13:20:17,748 CLibrary.java (line 67) JNA link failure details: /tmp/jna-oracle/jna1599621626582486116.tmp: /lib64/libc.so.6: version `GLIBC_2.11' not found (required by /tmp/jna-oracle/jna1599621626582486116.tmp) Now it was easy to fix the problem. There is a few solution^ 1) updated linux binaries for x86/amd64 against 2.1.3 and 2.2.5 2) Use JNA 3.3.0 or JNA 2.7.0 version you can download JNA 3.3.0 from the follow...

Apache pig provides a platform for analyzing very large data set. With apache pig you can easily analyze your data from Cassandra. Apache pig compiles instruction to sequences of Map-Reduce programs which will run on Hadoop cluster. Cassandra source provides a simple pig script to run pig with Cassandra data. Cassandra also provides CassandraStorage class which will load and store data from Cassandra DB, this class will no built in support for storing data with TTL (time to live). In many cases you have to update a few columns or rows with ttl to delete later automatically from DB. For that, i have patched the CassandraStorage class and add the similar functionality. Here is the patch Index: src/main/java/ru/atc/smev/cassandra/storage/CassandraStorage.java IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== --- src/main/java/ru/atc/smev/cassandra/storage/CassandraStorage.ja...