This week The apache Ignite book becomes one of the top books of leanpub

This week The apache Ignite book becomes one of the top books of leanpub.


An impatient start with Mahout - Machine learning

One of my friend ask me to develop a course for students in subject "Machine Learning". Course should be very simple to familiar students with machine learning. Main purpose of this course is to explorer the machine learning world to the students and playing over this topics with own their hands. Machine learning is very much matured and a lot of tools and frameworks is available to wet your hands in this topics, however most of the articles or tutorials you could found in the internet will start installing cluster or have to write a bunch of code (even, in site mahout, they are using maven) to start learning. Even more not all students are familiar with hadoop or do not have very powerful notebook to install and run all the components to get test of machine learning. For these reasons i have got the following approach:
  1. Standalone Hadoop
  2. Standalone Mahout 
  3. And a few CSV data files to learn how to works with Predictions

Assume you already have java installed in your work station. If not please refer to Oracle site to download and install java yourself. First we will install standalone hadoop, check the installation and  after that we will install Mahout and try to run some example to understand whats going under the hood. 
  • Install Hadoop and run simple map reduce for test
Hadoop version: 2.6.0 
Download hadoop-2.6.0.tar.gz from apache hadoop download site. In the moment of written the blog version 2.6.0 is stable to use. 
Unarchive the gz file some where in you local disk. Add these following path to your .bash_profile path as follows:
Now lets check the installation of the hadoop. Create one directory in current folder inputwords and copy all the xml files from the hadoop etc installation folder as follows:
$ mkdir inputwords
$ cp $HADOOP_HOME/etc/hadoop/*.xml inputwords/
Now we can run the hadoop standalone map reduce to count all the words found in the xmls
$ hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount inputwords outputwords
You should see a bunch of logs in your console and if everything went fine you should found a few files in folder outputwords (this folder will create in the runtime). Run the following command in you console
$cat ./outputwords/part-r-00000
it should show a lot of words follows by numbers as follows:
use 9
used 22
used. 1
user 40
user. 2
user? 1
users 21
users,wheel". 18
uses 2
using 3
value 19
values 1
version 1
version="1.0" 5
via 1
when 4
where 1
which 5
while 1
who 2
will 7
If you are curious to count more word, download the famous novel of William Shakespeare "The Tragedy of Romeo" from here and run with hadoop wordcount.
  • Download and install apache mahout 
Lets download the apache mahout distributive from here and unarchive the file some where in your local machine. Mahout distributive contains all the libraries and example for running machine learning  on top of hadoop. 
Now we need some data for learning purpose. We can use grouplens data for our purpose, certainly you can generate data for yourself, but i highly recommended data from grouplens. Grouplens organisation collecting social data for research and you can use these data for your purpose. There are a few datasets available in grouplens site such as MovieLens, BookCrosing and e.t.c. For my course we are going to use movielens datasets, because it's formatted and grouped well. Lets download the movielens datasets and unarchive the file somewhere in your filesystems.
First i would like to examine the datasets to get a closer look on the data, which will give us a very good understanding to use the data well. 
After unarchive the ml-data.tar.gz, you should find a list of datasets in your folder.     -- The full u data set, 100000 ratings by 943 users on 1682 items.
              Each user has rated at least 20 movies.  Users and items are
              numbered consecutively from 1.  The data is randomly
              ordered. This is a tab separated list of 
          user id | item id | rating | timestamp. 
              The time stamps are unix seconds since 1/1/1970 UTC     -- The number of users, items, and ratings in the u data set.

u.item     -- Information about the items (movies); this is a tab separated
              list of
              movie id | movie title | release date | video release date |
              IMDb URL | unknown | Action | Adventure | Animation |
              Children's | Comedy | Crime | Documentary | Drama | Fantasy |
              Film-Noir | Horror | Musical | Mystery | Romance | Sci-Fi |
              Thriller | War | Western |
              The last 19 fields are the genres, a 1 indicates the movie
              is of that genre, a 0 indicates it is not; movies can be in
              several genres at once.
              The movie ids are the ones used in the data set.

u.genre    -- A list of the genres.

u.user     -- Demographic information about the users; this is a tab
              separated list of
              user id | age | gender | occupation | zip code
              The user ids are the ones used in the data set.
For getting recommendation we will use Now it's times to study a few theory about Recommendation and what is for?

  • Recommendation:
Mahout contains a recommender engine—several types of them, in fact, beginning with conventional user-based and item-based recommenders. It includes implementations of several other algorithms as well, but for now we’ll explore a simple user-based recommender. For detail information of the recommender engine please see here.

  • Examine the DataSet:
Lets take a closer look to the file
196 242 3 881250949
186 302 3 891717742
22 377 1 878887116
244 51 2 880606923
166 346 1 886397596
298 474 4 884182806
115 265 2 881171488
253 465 5 891628467
305 451 3 886324817
6 86 3 883603013
62 257 2 879372434
286 1014 5 879781125
200 222 5 876042340
210 40 3 891035994
224 29 3 888104457
303 785 3 879485318
122 387 5 879270459
194 274 2 879539794
291 1042 4 874834944
234 1184 2 892079237
119 392 4 886176814
167 486 4 892738452
299 144 4 877881320
291 118 2 874833878
308 1 4 887736532
95 546 2 879196566
38 95 5 892430094
102 768 2 883748450
63 277 4 875747401
For example user with id 196 recommended film 242 with 3 preferences. I have import the file in excel and sort by userid as follows:

User 1 rate 3 films [1,2] and user 3 also rate film 2. If we want to find recommendation for user3 by user 1, then it should be the film 1 (Toy Story).
Lets run the recommendation engine of Mahout and examine the result.
$hadoop jar $MAHOUT_HOME/mahout-examples-0.10.0-job.jar -s SIMILARITY_COOCCURRENCE --input PATH_TO/ml-100k/ --output outputm
You should found the result in the file outputm/part-r-00000. If you check attentively, you should found that recommendation for the user 3 as follows:
3 [137:5.0,248:4.8714285,14:4.8153844,285:4.8153844,845:4.754717,124:4.7089553,319:4.7035174,508:4.7006173,150:4.68,311:4.6615386]
which is differ from that we guess earlier, because recommendation engine also use preferences (rating) from other user.
Lets write down a few bunch of code in java to make sure which films was recommended by Mahout.
package com.blu.mahout;

import org.apache.avro.generic.GenericData;

import java.nio.channels.Channels;
import java.nio.channels.FileChannel;
import java.nio.charset.Charset;
import java.nio.charset.CharsetDecoder;
import java.nio.charset.CodingErrorAction;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Optional;

import static;
import static;
import static;

 * Created by shamim on 17/04/15.
 * Source file should be,, u.item and mahout generated generated file
public class PrintRecomendation {
    private static final int INPUT_LENGTH = 6;

    private static List userRecommendedFilms = new ArrayList<>();
    private static List userWithFilmItems = new ArrayList<>();

    public static void main(String[] inputs) throws IOException{
        System.out.println("Print the recommendation and recommended films for given user:");

        if(inputs.length < INPUT_LENGTH){
                    "" + " Example: java -jar 3 u.item part-r-00000 /Users/shamim/Development/workshop/bigdata/hadoop/inputm/ml-100k/ /Users/shamim/Development/workshop/bigdata/hadoop/outputm/"
        String USER_ID = inputs[0];
        String UDATA_FILE_NAME = inputs[1];
        String UITEM_FILE_NAME = inputs[2];
        String MAHOUT_REC_FILE = inputs[3];
        String UDATA_FOLDER = inputs[4];
        String UMDATA_FOLDER = inputs[5];

        // Read UDATA File
        Path pathToUfile = Paths.get(UDATA_FOLDER,UDATA_FILE_NAME);
        List filteredLines = Files.lines(pathToUfile).filter(s -> s.contains(USER_ID)).collect(toList());
        for(String line : filteredLines){
            String[] words =  line.split("\\t");
            if(words != null){
                String userId = words[0];
        List userWithFilmName = new ArrayList();

        CharsetDecoder dec= StandardCharsets.UTF_8.newDecoder()
        Path pathToUItem=Paths.get(UDATA_FOLDER, UITEM_FILE_NAME);

        List nonFiltered;

        try(Reader r= Channels.newReader(, dec, -1);
            BufferedReader br=new BufferedReader(r)) {

        Path pathToMahoutFile = Paths.get(UMDATA_FOLDER, MAHOUT_REC_FILE);

        List filteredMLines = Files.lines(pathToMahoutFile).filter(s -> s.contains(USER_ID)).collect(toList());
        String recommendedFilms= "";
        for(String line : filteredMLines){
            String[] splited = line.split("\\t");
                recommendedFilms = splited[1];

        String[] filmsId = recommendedFilms.split(",");

        for(String filmId : filmsId){
            String[] idWithRating = filmId.split(":");
            String id = idWithRating[0];
            String rating = idWithRating[1];
            for(String filmLine : nonFiltered ){
                String[] items = filmLine.split("\\|");
                    System.out.println("Film name:" + items[1]);


You should found the following output in your console:
Film name:Grosse Pointe Blank (1997)
Film name:Postino, Il (1994)
Film name:Secrets & Lies (1996)
Film name:That Thing You Do! (1996)
Film name:Lone Star (1996)
Film name:Everyone Says I Love You (1996)
Film name:People vs. Larry Flynt, The (1996)
Film name:Swingers (1996)
Film name:Wings of the Dove, The (1997)

No comments :