|本期目录/Table of Contents|

[1]刘君.基于Hadoop的海量小文件存储优化方法[J].厦门理工学院学报,2017,(3):34-39.
 LIU Jun.An Storage Optimization Method for Storage of MassiveSmall Files Based on Hadoop[J].Journal of JOURNAL OF XIAMEN,2017,(3):34-39.
点击复制

基于Hadoop的海量小文件存储优化方法(PDF)
分享到:

《厦门理工学院学报》[ISSN:1673-4432/CN:35-1289/Z]

卷:
期数:
2017年第3期
页码:
34-39
栏目:
计算机与信息工程
出版日期:
2017-06-30

文章信息/Info

Title:
An Storage Optimization Method for Storage of Massive Small Files Based on Hadoop
文章编号:
1673-4432(2017)03-0034-06
作者:
刘君
(潍坊科技学院中印计算机软件学院,山东 潍坊 262700)
Author(s):
LIU Jun
(College of SinoIndia Computer Software,Weifang University of Science and Technology,Weifang 262700,China)
关键词:
小文件存储Hadoop小文件归并预取缓存
Keywords:
small file storageHadoopmerged small filesprefetchcache
分类号:
TP333
DOI:
-
文献标志码:
A
摘要:
基于Hadoop的海量小文件存储进行优化,利用小文件内部存在的相互联系,进行小文件的归并操作;通过索引机制访问小文件及元数据缓存,并利用相关性强的小文件预取机制提高文件的读取效率。实验发现,优化后的方法降低了Hadoop名字节点的内存消耗,减少了查询时耗,提高了系统性能。
Abstract:
An optimization method for storage of massive small files based on Hadoop was proposed,using the precursor of subsequent relationship between different education resource in small files to merge small files,accessing small files and caching metadata by index mechanism,using the prefetching mechanism of associated small files to improve reading efficiency.Experimental results show that this method reduces the memory consumption of the Hadoop name node and the query time,and improves the performance of the system.

参考文献/References:

[1]陆嘉恒.Hadoop实战[M].北京:机械工业出版社,2012. [2]BORTHAKUR D.The Hadoop distributed file system:architecture and design[J].Hadoop Project Website,2007,11(11):110. [3]WHITE Tom.The small files problem[EB/OL].[20090202].http://www.cloudera.com/blog/2009/02/thesmallfilesproblem. [4]DONG Bo,QIU Jie,ZHENG Qinghua,et al.A novel approach to improving the efficiency of storing and accessing small files on Hadoop:a case study by powerpoint files[C]//IEEE International Conference on Services Computing.Piscataway:IEEE,2010:6572. [5]李路杰.Hadoop中小文件处理技术的研究与优化[D].石家庄:河北大学,2011. [6]张守利,杨冬菊,韩燕波.一种面向海量小文件的文件接收和存储优化方案[J].小型微型计算机系统,2015,36(8):1 7471 751. [7]丁建立,郑峰弓,李永华,等.基于NoSQL的海量航空物流小文件分布式多级存储方法[J].计算机应用研究,2017,34(5):1 4331 441. [8]LI Jia,LIN Kunhui,WANG Jingjin.Design of the mass multimedia files storage architecture based on Hadoop[C]//The 8th International Conference on Computer Science & Education.Piscataway:IEEE,2013:801804. [9]王涛,姚世红,徐正全,等.云存储中面向访问任务的小文件归并与预取策略[J].武汉大学学报(信息科学版),2013,38(12):1 5041 508. [10]CHANDRASEKAR S,DAKSHINAMURTHY R,SESHAKUMAR P G,et al.A novel indexing scheme for efficient handling of small files in hadoop distributed file system[C]//2013 International Conference on Computer Communication and Informatics (ICCCI).Piscataway:IEEE,2013:18. [11]钱能武,郭卫斌,范贵生.基于关联规则挖掘的分布式小文件存储方法[J].华东理工大学学报(自然科学版),2016(5):708714.

相似文献/References:

备注/Memo

备注/Memo:
[收稿日期]2017-05-22[修回日期]2017-06-02 [基金项目]潍坊科技学院校级课题(W14K027) [作者简介]刘君(1986-),女,助教,硕士,研究方向为计算机应用技术,Email:731083964@qq.com。
更新日期/Last Update: