My workspace

Apache Ignite is widely used around the world and is growing all the time. Companies like Barclays, Misys, Sberbank (3r largest bank in Europe), ING, JacTravel all use Ignite to power pieces of their architecture that are critical to the day-to-day operations of those organizations. Moreover, the vendor like TIBCO uses core caching data-grid module of Apache Ignite with advanced indexing and SQL capability for their Master Data Management platform. However there are a few others alternatives to Apache Ignite from other vendors such as HazelCast, Oracle, Ehcache, GemFire, etc. The main difference of Apache Ignite from the others is the number of functionalities and simplicity of use. Apache Ignite provides a variety of functionalities, which you can use for different use cases. The key differences between the Apache Ignite , Hazelcast , and Apache Cassandra are as follows: Feature Apache Ignite Hazelcast Apache Cassandra Data model Key-value ...

We are happy to introducing our new title " The Apache Ignite book ". The first portion of the book is being published a few days before by leanpub publication. Note that, the entire book is an agile published book, which means that the authors are releasing it as they write it and will continue to update the book as the underlying technology evolves over time. The sample chapter of the book is available in different formats including HTML . What you will learn: Apache Ignite architecture in depth such as data distributing technics (DHT), Rendezvous hashing, memory architecture, various topology and caching strategies. Apache Ignite use cases as a memory-centric distributed database, caching and computing platforms. Caching strategies and how to use Apache Ignite to improve the performance of the application. Using JPA with Apache Ignite for developing high-performance web applications . How to accelerates the performance of your existing Hadoop ecosystem withou...

Yesterday i visited Saint Petersburg university after a long period of time as a speaker. Conference held already a weeks and students came here from all over the country and out side of Russia. Audiences was very nice and ask a lot of technical question mostly about Cassandra and we are utilising it in our projects. Community was very curious about NoSQL and wanted find all the answer of their questions about the new trend. Here is the presentation from slide share net. NoSQL - World IT Planet, Saint Petersburg 2015 from Shamim bhuiyan

Offtopic: My first book "High performance in-memory computing with Apache Ignite" has been released and available at http://leanpub.com/ignite . This post is my review of the Packt Publishing book Cassandra High Availability by Robbie Strickland. As the main title suggest. The book has almost 186 pages covering 9 chapters and must read for Cassandra users. Chapter 1: Cassandra's approach to High Availability The initial chapter of Cassandra High Availability cover some architectural design such as Monolithic Architecture, Master slave Architecture and the Cassandra's approach to achieve high availability. Most of all modern software system requires a non functional requirements such as High availability. In this chapter author briefly describe why RDBMS is the single point of failure and is not suitable for horizontal scaling, also the drawback of the Master Slave architecture. In the next few section Author describe the Cassandra's architecture and it's...

This post is my review of the Packt Publishing book Cassandra Design patterns by Sanjay Sharma. As the main title suggest, it's all about pattern and anti pattern of using Cassandra. The book has almost 74 pages covering 6 chapters. Preface: What this book covers and Who this book is for The preface of the book starts with the ideas behind this book. The main idea behind this book is for Cassandra audience to understand where and how to use Cassandra correctly and effectively.The Preface also provides brief summaries of each of the six chapters in the book and convention that follows in this books. Under the section "What you need for this book", author specified that, readers don't need any special version of Cassandra, however Cassandra 2.0 or above version will be proffered. The "Who this book is for" section of the Preface specify the audience of the book, it may be architect, or developer starting with Cassandra. Chapter 1: An Overview of Architect...

Sooner or later every enterprise application needs full text search with their content. Slor, elasticsearch based on lucene are one the best candidate for developying enterprise search. Elasticsearch got very popularity with its simplicity, but out of box it dosen't support importing data from Cassandra cluster. However Elasticsearch provides river, a river is a pluggable service running within elasticsearch cluster pulling data (or being pushed with data) that is then indexed into the cluster. With a few search i have found a cassandra-river on github from ebay, unfortunatley, project was legeacy and only support Cassandra version 1.2*. With a few effort i rewrite the project with data stax cassandra driver. Here you can find the project, now it support the following features: 1) Cron scheduling; 2) Reading Cassandra rows through Paging; 3) Based on DataStax java driver 2.0; For quick installation, download the project from the Github. Build with maven: mvn clean install i...

Interview on PlanetCassandra.

A few days ago I attended in Moscow Cassandra meet up with my presentation, from one of the participant, I heard about Facebook project presto for fast data analysis. I was very curious and hurry up to hands on it. From Presto Site "Presto is a distributed SQL query engine optimized for ad-hoc analysis at interactive speed. It supports standard ANSI SQL, including complex queries, aggregations, joins, and window functions". Historically Cassandra was lack of interactive Ad-hoc query, even it's doesn't support any aggregate function in CQL. For this reason, whenever we proposed our customers to utilize Cassandra as a database, they were always confused. However, for analysis data over Cassandra we have the following frameworks: 1) Hadoop Map Reduce 2) Spark and Shark Also a few commercial projects like impala. But Hadoop Map Reduce is definitely slow to use as Ad-Hoc queries. Spark is very fast with its RDD data models, but it also needs a few exercises to run q...

The last few months I was busy with our new telecommunication project to develop MNP (Mobile number portability) for the Russian Federation. Now in Russia anybody can change their telephone operator without changing the number, it's a another history for another blog. Today I have found a few hours to keep my promise. In this blog, I will try to describe how to configure and manage spark pseudo cluster with shark for real time data processing. In the previous blog I will show how to use hive with Hadoop to process data from Cassandra. For whom, who doesn't familiar, Spark is execution engines that supports cyclic data flow and in-memory computing, in the otherhand Shark is an open source distributed SQL query engine for Hadoop data. It brings state-of-the-art performance and advanced analytics to Hive users. I am going to use following open source projects to configure and run the spark + shark cluster : 1) Scala-2.10.3 2) Spark-0.9.0-incubating-bin-hadoop1 3) Shark-0.9.0...

This is the first part of getting start with real time data processing with Cassandra. In the first part i am going to describe how to configure Hadoop, Hive and Cassandra, also some adhoc query to use new CqlStorageHandler. In the second part i will show, how to use Shark and Spark for real time fast data processing with Cassandra. I was encourage by the blog from the Data Stax, you can find out it here . Also all the credit goes for the author of the library cassandra-handler and Alex Lui for developing the CQLCassandraStorage. Of course you can use DataStax enterprise version for the first part, Data Stax enterprise version has built in support Hive and Hadoop. In this blog post i will use all the native apache products. If you are interested in Real time data process, please check this blog . In the first part i will use following products: 1) Hadoop 1.2.1 (Single node cluster) 2) Hive 0.9.0 3) Cassandra 1.2.6 (Single node cluster) 4) cassandra-handler 1.2.6 (depends on Hive...

Big data Moscow meetup presentation, described our use case, problems we solved and how we utilized Cassandra and Hadoop with Big data processing Big data moscow meetup from shamim_ru

One of the main disadvantage of using PIG is that, Pig always raise all the data from Cassandra Storage, and after that it can filter by your choose. It's very easy to imagine how the workload will be if you have a tons of million rows in your CF. For example, in our production environment we have always more than 300 million rows, where only 20-25 millions of rows is unprocessed. When we are executing pig script, we have got more than 5000 map tasks with all the 300 millions of rows. It's time consuming and high load batch processing we always tried to avoid but in vain. It's could be very nice if we could use CQL query in pig scripts with where clause to select and filter our data. Here benefit is clear, less data will consume, less map task and a little workload. Still in latest version of Cassandra (1.2.6) this feature is not available. This feature is planned in next version Cassandra 1.2.7. However patch is already available for this feature, with a few efforts we ...

After a few weeks break at last we completed our tuning and configuration cassandra hadoop stack in production. It was exciting and i decided to share our experience with all. 1) Cassandra version >> 1.2 has some problems and doesn't integrate with Hadoop very well. The problem with Map Reduce, when we runs any Map reduce job, it always assigns only one mapper regardless of the amount of data. See here for more detail. 2) If you are going to use Pig for you data analysis, think twice, because Pig always picks up all the data from the Cassandra Storage and only after these it can filter. If you have a billions of rows and only a few millions of then you have to aggregate, then Pig always pick up the billions of rows. Here you can find a compression between Hadoop framework for executing Map reduce. 3) If you are using Pig, filter rows as early as possible. Filter fields like null or empty. 4) When using Pig, try to model your CF slightly different. Use Bucket pattern, sto...

Last couple of months we are using Apache Hadoop Map Reduce batch processing to analyze a huge amount of data. We have a few legacy product where we can't consider to using Cassandra big table database. A few of them uses Oracle Database as their primary storage. As our requirements we have to extract the data from the rdbms, parse the payload and load it to Cassandra for aggregating. Here i have decided to use Storm for real time data processing. Our usecase is as follows: 1) Storm spout connects to Oracle Database and collects data from particular table with some intervals. 2) Storm bolt parses the data with some fashion and emit to Storm-cassandra bolt to store the row into Cassandra DB. Here is the fragment code of project. First i have create a Jdbc connector class, class contain a few class variables which contradicting with Storm ideology, as far i have just need one spout as input - it's enough for me. package storm.contrib.jdbc; import org.slf4j.Logger; import ...

Today our team tried to install JNA 3.5.0 in our UAT environment. Here is the link to make a try. At the moment of Cassandra start we have noticed the following INFO on cassandra log: INFO [main] 2012-11-27 13:20:17,747 CLibrary.java (line 66) JNA link failure, one or more native method will be unavailable. Very interesting thing is that, most of the JNA features works. I have decided to investigate the problem and restart Cassandra in debug mode (edit the log4j-server.properties and set the rootLogger level DEBUG) and found the details error DEBUG [main] 2012-11-27 13:20:17,748 CLibrary.java (line 67) JNA link failure details: /tmp/jna-oracle/jna1599621626582486116.tmp: /lib64/libc.so.6: version `GLIBC_2.11' not found (required by /tmp/jna-oracle/jna1599621626582486116.tmp) Now it was easy to fix the problem. There is a few solution^ 1) updated linux binaries for x86/amd64 against 2.1.3 and 2.2.5 2) Use JNA 3.3.0 or JNA 2.7.0 version you can download JNA 3.3.0 from the follow...

Apache pig provides a platform for analyzing very large data set. With apache pig you can easily analyze your data from Cassandra. Apache pig compiles instruction to sequences of Map-Reduce programs which will run on Hadoop cluster. Cassandra source provides a simple pig script to run pig with Cassandra data. Cassandra also provides CassandraStorage class which will load and store data from Cassandra DB, this class will no built in support for storing data with TTL (time to live). In many cases you have to update a few columns or rows with ttl to delete later automatically from DB. For that, i have patched the CassandraStorage class and add the similar functionality. Here is the patch Index: src/main/java/ru/atc/smev/cassandra/storage/CassandraStorage.java IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== --- src/main/java/ru/atc/smev/cassandra/storage/CassandraStorage.ja...

Recently i have found one interesting project on Github named PlayOrm , which features very impress me. I have decided to just play with it. Lets first check out there features list: Just added support for Entity has a Cursor instead of List which is lazy read to prevent out of memory on VERY wide rows PlayOrm Queries use way less resources from cassandra cluster than CQL queries Scalabla JQL(SJQL) supported which is modified JQL that scales(SQL doesn't scale well) Partitioning so you can query a one trillion row table in just ms with SJQL(Scalable Java Query Language) Typical query support of <=, <, >, >= and = and no limitations here Typical query support of AND and OR as well as parenthesis Inner Join support (Must keep your very very large tables partitioned so you get very fast access times here) Left Outer Join support Return Database cursor on query OneToMany, ManyToMany, OneToOne, and ManyToOne but the ToMany's are nosql fashion not like RDBMS ...