Mahout 是 Apache Software Foundation(ASF) 旗下的一个开源项目,提供一些可扩展 的机器学习领域经典算法的实现,旨在帮助开发人员更加方便快捷地创建智能应用程序。
Mahout包含许多实现,包括聚类、分类、推荐过滤、频繁子项挖掘。此外,通过使用 Apache Hadoop 库,Mahout 可以有效地扩展到云中。

一、maven构建mahout开发环境

开发环境
$ mvn -v
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19
21:51:
28+0800)
Maven home: d:\software\apache\apache-maven-3.0.5
Java version: 1.7.0_67, vendor: Oracle Corporation
Java home: d:\software\Java\jdk1.7.0_67\jre
Default locale: zh_CN, platform encoding: GBK
OS name: “windows 8.1”, version: “6.3”, arch: “amd64”, family: “windows”

1、mvn命令生成标准java工程

1
2
$ mvn archetype:generate -DarchetypeGroupId=org.apache.maven.archetypes 
-DgroupId=cn.itweet.mymahout -DartifactId=myMahout -DpackageName=cn.itweet.mymahout -Dversion=1.0-SNAPSHOT -DinteractiveMode=false
1
2
$ ls myMahout/
pom.xml src

2、把生成的java项目导入eclipse

1
选择系统1.7jdk,以及maven插件,还有字符编码为utf-8

3、增加mahout依赖,修改pom.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<modelVersion>4.0.0</modelVersion>
<groupId>cn.itweet.mymahout</groupId>
<artifactId>myMahout</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<name>myMahout</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<mahout.version>0.6</mahout.version>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.mahout</groupId>
<artifactId>mahout-core</artifactId>
<version>${mahout.version}</version>
</dependency>
<dependency>
<groupId>org.apache.mahout</groupId>
<artifactId>mahout-integration</artifactId>
<version>${mahout.version}</version>
<exclusions>
<exclusion>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.cassandra</groupId>
<artifactId>cassandra-all</artifactId>
</exclusion>
<exclusion>
<groupId>me.prettyprint</groupId>
<artifactId>hector-core</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</project>

4、下载依赖

1
2
$ cd myMahout/
$ mvn clean install

二、用Mahout实现协同过滤userCF

1、manhout推荐引擎源码

1
2
3
4
5
6
7
8
9
10
Mahout的推荐引擎,要从org.apache.mahout.cf.taste包说起。
common: 公共类包括,异常,数据刷新接口,权重常量
eval: 定义构造器接口,类似于工厂模式
model: 定义数据模型接口
neighborhood: 定义近邻算法的接口
recommender: 定义推荐算法的接口
similarity: 定义相似度算法的接口
transforms: 定义数据转换的接口
hadoop: 基于hadoop的分步式算法的实现类
impl: 单机内存算法实现类

2、程序开发: Mahout推荐API

1
2
3
4
   org.apache.mahout.cf.taste.model.DataModel.java
 org.apache.mahout.cf.taste.similarity. UserSimilarity.java
 org.apache.mahout.cf.taste.impl.neighborhood.NearestNUserNeighborhood.java
 org.apache.mahout.cf.taste.recommender.Recommender.java

3、demo

3.1 项目结构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
D:\software\ProjectSource\scala_ws\myMahout>tree
D:.
├─.settings
├─datafile
├─src
│ ├─main
│ │ └─java
│ │ └─cn
│ │ └─itweet
│ │ └─mymahout

│ └─test
│ └─java
│ └─cn
│ └─itweet
│ └─mymahout

└─target
├─classes
│ └─cn
│ └─itweet
│ └─mymahout

├─maven-archiver
├─surefire
├─surefire-reports
└─test-classes
└─cn
└─itweet
└─mymahout

3.2 数据

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
1,101,5.0
1,102,3.0
1,103,2.5
2,101,2.0
2,102,2.5
2,103,5.0
2,104,2.0
3,101,2.5
3,104,4.0
3,105,4.5
3,107,5.0
4,101,5.0
4,103,3.0
4,104,4.5
4,106,4.0
5,101,4.0
5,102,3.0
5,103,2.0
5,104,4.0
5,105,3.5
5,106,4.0

据解释:每一行有三列,第一列是用户ID,第二列是物品ID,第三列是用户对物品的打分。

运行结果:
uid:1(104,4.274336)(106,4.000000) uid:2(105,4.055916) uid:3(103,3.360987)(102,2.773169) uid:4(102,3.000000) uid:5

3.2 程序

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
package cn.itweet.mymahout;

import java.io.File;
import java.io.IOException;
import java.util.List;

import org.apache.mahout.cf.taste.common.TasteException;
import org.apache.mahout.cf.taste.impl.common.LongPrimitiveIterator;
import org.apache.mahout.cf.taste.impl.model.file.FileDataModel;
import org.apache.mahout.cf.taste.impl.neighborhood.NearestNUserNeighborhood;
import org.apache.mahout.cf.taste.impl.recommender.GenericUserBasedRecommender;
import org.apache.mahout.cf.taste.impl.similarity.EuclideanDistanceSimilarity;
import org.apache.mahout.cf.taste.model.DataModel;
import org.apache.mahout.cf.taste.recommender.RecommendedItem;
import org.apache.mahout.cf.taste.recommender.Recommender;
import org.apache.mahout.cf.taste.similarity.UserSimilarity;


/**
* Created by root on 2015年7月17日
*/

public class UserCF {

final static int NEIGHBORHOOD_NUM = 2;
final static int RECOMMENDER_NUM = 3;

public static void main(String[] args) throws IOException, TasteException {
String file = "datafile/item.csv";
DataModel model = new FileDataModel(new File(file));
UserSimilarity user = new EuclideanDistanceSimilarity(model);
NearestNUserNeighborhood neighbor = new NearestNUserNeighborhood(NEIGHBORHOOD_NUM, user, model);
Recommender r = new GenericUserBasedRecommender(model, neighbor, user);
LongPrimitiveIterator iter = model.getUserIDs();

while (iter.hasNext()) {
long uid = iter.nextLong();
List<RecommendedItem> list = r.recommend(uid, RECOMMENDER_NUM);
System.out.printf("uid:%s", uid);
for (RecommendedItem ritem : list) {
System.out.printf("(%s,%f)", ritem.getItemID(), ritem.getValue());
}
System.out.println();
}
}
}

三、 用Mahout实现推荐系统

1、开发前的规划

1
2
3
 BookEvaluator.java, 选出“评估推荐器”验证得分较高的算法
 BookResult.java, 对指定数量的结果人工比较
 BookFilterGenderResult.java,只保留男性用户的图书列表

2、人工对数据进行分析

根据最后计算结果去核对,推荐结果是否符合要求!或者符合自然现象.
推荐是否合理。通过比对找到一些规则去排除数据,让推荐系统更加准确!
利用可视化软件帮助完成分析过程!

四、mahout-kmeans聚类测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
http://mahout.apache.org/users/clustering/k-means-clustering.html

# wget http://archive.ics.uci.edu/ml/databases/synthetic_control/synthetic_cont
rol.data

# mahout org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
error:Input path does not exist: hdfs://server1:8020/user/hive/testdata

$ hadoop fs -mkdir hdfs://server1:8020/user/hive/testdata

# sudo -u hdfs hadoop fs -put synthetic_control.data /user/hive/testdata

-- 查看结果
$ hadoop fs -ls hdfs://server1:8020/user/hive/output/data
-rw-r--r-- 3 hive hdfs 335470 2015-09-03 01:07 hdfs://server1:8020/user/hive/output/data/part-m-00000

$ mahout vectordump -i ./output/data/part-m-00000
15/09/03 01:17:00 INFO common.AbstractJob: Command line arguments: {--endPhase=[2147483647], --input=[./output/data/part-m-00000], --startPhase=[0], --tempDir=[temp]}
15/09/03 01:17:03 INFO vectors.VectorDumper: Sort? false

{0:28.7812,31:26.6311,34:29.1495,4:28.9207,32:35.6541,5:33.7596,8:35.2479,6:25.3969,30:25.0293,24:33.0292,29:34.9424,17:26.5235,51:24.5556,36:26.1927,12:36.0253,23:29.5054,58:25.4652,21:29.27,11:29.2171,10:32.8717,15:32.8717,7:27.7849,28:26.1203,46:28.0721,33:28.4353,55:34.9879,54:34.9318,25:25.04,3:31.2834,49:29.747,41:26.2353,1:34.4632,26:28.9167,44:31.0558,37:33.3182,56:32.4721,42:28.9964,27:24.3437,50:31.4333,16:34.1173,40:35.5344,48:35.4973,39:27.0443,9:27.1159,52:33.7431,13:32.337,43:32.0036,19:26.3693,59:25.8717,2:31.3381,20:25.7744,18:27.6623,22:30.7326,35:28.1584,57:33.3759,45:34.2553,38:30.9772,47:28.9402,14:34.5249,53:25.0466}
{0:24.8923,31:32.5981,34:26.9414,4:27.8789,32:28.3038,5:31.5926,8:27.9516,6:31.4861,30:34.0765,24:31.9874,29:25.0701,17:35.6273,51:31.0205,36:33.1089,12:27.4867,23:30.4719,58:32.1005,21:24.1311,11:31.1887,10:27.5415,15:24.488,7:35.5469,28:33.6472,46:26.3458,33:26.1471,55:26.4244,54:33.6564,25:33.6615,3:32.8217,49:29.4047,41:26.5301,1:25.741,26:25.5511,44:32.8357,37:24.1491,56:28.4661,42:24.8578,27:30.4686,50:32.5577,16:27.5918,40:35.9519,48:28.9861,39:25.7906,9:31.6595,52:26.6418,13:31.391,43:25.9562,19:31.4167,59:26.691,2:27.5532,20:30.7447,18:35.4102,22:35.1422,35:31.5203,57:34.2484,45:28.5322,38:28.5157,47:30.6213,14:27.811,53:28.4331}
......

参考:http://blog.fens.me/hadoop-mahout-maven-eclipse/

http://blog.fens.me/mahout-recommend-engine/