摘要
        本文提出了一种基于Hadoop平台的分布式数据存储和查询改进方案。本文重点阐述了通信数据分布式存储与查询在Hadoop平台上的实现。据Hbase数据库的特色设计了通信社会网络数据模型。实现了基于Hbase数据库上的条件查询。并对设计好的数据模型进行了优化。最后在Hadoop数据服务平台上实现了面向软件客户端的数据访问服务。设计了适合通信社会网络数据特色的Map／Reduce算法。通过Map和Reduce函数实现了数据的并行查询和处理。在数据查询过程中还将数据遍历过程放在Reduce函数中从而使广度优先搜索算法的层次遍历过程也能够并行运行。这在很大程度上优化了数据查询和分层扩展的效率。通信数据分布式存储与查询方案在Hadoop平台上的成功实施具有很重要的意
义。Hadoop平台只需要部署在普通的廉价PC机上即可运行，却能够处理PB的级
数据。高效却又节约成本，因此具有很高的现实意义与应用价值。
 
关键词：Hadoop平台；Map／Reduce算法；分布式查洵；Hbase
 

Abstract
After analyzing the existing distributed storage systems and cloud computing platform，we choose Hadoop platform for distributed data storage and query to improve the program．The papers focus on the communication data distributed storage and query based on Hadoop platform．Design the Hbase．based communication data model of social networkdata．We implement the conditions query．Design and optimize data model．Finally,theclients Can access services from Hadoop platform．Design and implement Map／Reducealgorithm for communication data set．Map and Reduce functions implement the dataparallel query processing．In the data query process，the traverse process is put in theReduce function．SO that the BFS algorithm traverse can also run in parallel．This is inlarge measure to optimize the data query and the efficiency of stratification expansion．The implementation of communication data distributed storage and query based onHadoop platform has very important significance．Hadoop platform needs to bedeployed only in the ordinary，cheap PC to run，but able to deal with the PB data．Efficient but cost．effective，it has high value and application of practical significance．
KEYWORDS：Hadoop platform；Map／Reduce algorithm；Distributed query；Hbase
 

目录
1绪论......................................................................................................................................... 1
1．1 课题研究背景............................................................................................................. 1
1．2课题研究内容与意义................................................................................................... 1
2 Hadood平台架构..................................................................................................................... 4
2．1 Hadoop技术背景......................................................................................................... 4
2．1．1 l BM的云计算................................................................................................ 4
2．1．2 Goog I e文件系统GFS.................................................................................... 4
2．1．3 Map／Reduce算法.......................................................................................... 6
2．2 Hadoop平台概述......................................................................................................... 7
2．3．1 假定与目标.................................................................................................... 9
2．3．2 名字节点与数据节点.................................................................................... 10
2．3．3 数据复制...................................................................................................... 10
2．3．4 数据组织...................................................................................................... 11
2．4 Map／Reduce算法实现............................................................................................. 11
2．4．1 主要操作实现............................................................................................... 11
2．4．2 Map／Reduce执行流程................................................................................. 12
2．4．3 其他关键技术............................................................................................... 14
2．5 本章总结.................................................................................................................. 14
3通信数据模型设计................................................................................................................. 15
3．1 Google的BigTable.................................................................................................... 15
3．2 Hbase体系结构......................................................................................................... 15
3．2．1 Hreg i on服务器............................................................................................ 16
3．2．2 主服务器...................................................................................................... 17
3．3 Hbase数据模型......................................................................................................... 17
3．4通信数据模型设计..................................................................................................... 18
3．4．1 数据模型设计分析........................................................................................ 18
3．4．2 主表设计...................................................................................................... 19
3．5 元数据模型设计........................................................................................................ 21
3．6 条件查询优化........................................................................................................... 23
3．7 数据存储性能优化.................................................................................................... 23
3．8 本章总结.................................................................................................................. 23
4分布式查询算法设计.............................................................................................................. 24
4．1 传统扩展查询算法.................................................................................................... 24
4．1．1 扩展查询算法概述........................................................................................ 24
4．1．2 存在问题...................................................................................................... 25
4．2 扩展算法的Hadoop实现........................................................................................... 25
4．2．1设计思想....................................................................................................... 25
4．2．3 Comb i Re函数设计...................................................................................... 27
4．2．4 Reduce函数设计........................................................................................... 27
4．2．5 Run函数设计................................................................................................ 27
4．3 Map／Reduce相关配置............................................................................................. 28
4．3．1 任务粒度...................................................................................................... 28
4．3．2 系统容错机制............................................................................................... 28
4．3．3 分类函数...................................................................................................... 29
4．4 两种系统实现比较.................................................................................................... 29
4．5 本章总结........................................................................................................... 29
5系统实施................................................................................................................................ 30
5．1 方案部署.................................................................................................................. 30
5．1．1 硬件环境配置............................................................................................... 30
5．1．2 SSH配置....................................................................................................... 30
5．1．3 数据节点和名字节点设置............................................................................. 30
5．1．4 Hadoop平台搭建........................................................................................... 31
5．1．5 Hbase环境搭建............................................................................................. 31
5．1．6 与客户端系统结合........................................................................................ 31
5．2 运行结果与分析........................................................................................................ 32
5．2．1 不同数据规模的实验.................................................................................... 32
5．2．2 不同任务粒度的实验.................................................................................... 33
5．3 本章总结........................................................................................................... 34
6总结与展望............................................................................................................................ 35
6．1论文总结................................................................................................................... 35
6．2论文展望................................................................................................................... 36
参考文献.................................................................................................................................. 36