赢发彩票代理

上海代写论文网专业提供代写毕业论文、代写本科论文服务
国内外关于大数据技术的研究现状
发布时间:2015-10-21
摘要

  0 YIN YAN

赢发彩票代理   SUIZHEJISUANJIJISHUHEHULIANWANGDEJISUFAZHAN,TEBIESHISUIZHE Web2. 0DEFAZHAN,HULIANWANGSHANGDESHUJULIANGGAOSUZENGZHANG,DUIDASHUJUCHULINENGLIXIANGDUIBUZU。BANSUIZHEDAICHULISHUJUYUELAIYUEDUO,DANGQIANYIJINGDAOLEWUFAJIANGDASHUJUCUNCHUZAIYITAIHUOYOUXIANSHUMUDEFUWUQINEI,GENGWUFAYOUSHUMUYOUXIANDEJISUANJILAICHULIDASHUJUDEJIONGJING。YINCI,RUHESHIXIANZIYUANHEJISUANNENGLIDEFENBUSHIGONGXIANGYIJIRUHEYINGDUIDANGQIANSHUJULIANGGAOSUZENGZHANGDESHITOU,SHIMUQIANSHUJUGUANLI、SHUJUCHULILINGYUJIDAIJIEJUEDEWENTI。

  IBM JICHULEDASHUJUDESANGE V,FENBIESHI: Volume,Variety,Ve-locity.DASHUJUZAI Volume TIXIANDESHICHULIDESHUJULIANGZHENGCONG T JIBIEKUAYUEDAO Z JIBIE,ZAI Variety TIXIANDESHICHULIDESHUJULEIXINGCONGJIEGOUHUALEIXINGWEIZHUGUODUDAOJIEGOUHUAHEFEIJIEGOULIANGZHONGSHUJULEIXINGGONGCUN; ZAI Velocity TIXIANDESHIDUISHUJUCHULIDEXIANGYINGSHIJIANCONGPICHULIXIANGYINGSHIJIANDAOSHISHIDELIUSHUJUCHULIXIANGYINGSHIJIAN。BENWENJICHUDASHUJUDEMIAOSHU: GUIMODA; SHENDUDA; KUANDUDA; CHULISHIJIANDUAN; YINGJIANXITONGPUTONGHUA; RUANJIANXITONGKAIYUANHUA。

赢发彩票代理   1 GUONEIWAIYANJIUXIANZHUANG

  1. 1 BINGXINGSHUJUKUYANJIU

  DASHUJUCHULIDECHUANTONGFANGFASHISHIYONGBINGXINGSHUJUKUXITONG。BINGXINGSHUJUKUXITONGSHIZAIDAGUIMOBINGXINGCHULIXITONG( MPP) HEJIQUNBINGXINGJISUANHUANJINGDEJICHUSHANGJIANLIDEGAOXINGNENGSHUJUKUXITONG。ZHEIYANGDEXITONGSHIYOUXUDUOSONGOUHECHULIDANYUANZUCHENGDE,ZHIDESHICHULIDANYUANERBUSHICHULIQI。MEIGEDANYUANNEIDE CPU DOUYOUZIJISIYOUDEZIYUAN,RUZONGXIAN、NEICUN、YINGPANDENG。ZAIMEIGEDANYUANNEIDOUYOUCAOZUOXITONGHEGUANLISHUJUKUDESHILIFUBEN。ZHEIZHONGJIEGOUZUIDADETEDIANZAIYUBUGONGXIANGZIYUAN。GUOWAIZAIGAILINGYUYANJIUQIBUJIAOZAO,ZAI 20 SHIJI 80 NIANDAIJIUCHUXIANLEKEYIYUNXINGZAIFEIGONGXIANGJIEDIANZUCHENGDEJIQUNSHANGDESHUJUKUXITONG。ZHEIXIEXITONGDOUZHICHIBIAOZHUNDEGUANXIXINGBIAOHE SQL,TONGSHIDUIYUZHONGDUANYONGHULAISHUOSHUJUSHISHISHANGSHITOUMINGDICUNCHUZAIDUOGEJIQISHANGDE。HENDUOZHEILEIXITONGDOUSHIJIANLIZAI Gamma[4]HE Grace[5]DEBINGXING DBMS XIANGMUDEXIANQUXINGYANJIUCHENGGUOZHISHANGDE。80 NIANDAIHOUQI,BINGXINGSHUJUKUJISHUDEYANJIUFANGXIANGZHUBUZHUANDAOLETONGYONGBINGXINGJIFANGMIAN,YANJIUDEZHONGDIANSHIBINGXINGSHUJUKUDEWULIZUZHI、CAOZUOSUANFA、YOUHUAHEDIAODUCELUO。

赢发彩票代理   ZAOZAI 20 SHIJI 80 NIANDAIZHONGQI,Teradata HE Gamma XIANGMUJIUKAISHITANSUOYIZHONGJIYUTONGGUOGAOSUHULIANWANGLUOLIANJIEDE“share-nothing JIEDIAN”( JUYOUDULI CPU,ZHUCUNHECIPAN) ZUCHENGDEJIQUNDEQUANXINBINGXINGSHUJUKUJIAGOUMOSHI[30],RUTU 1 SUOSHI。CONGNEISHIQI,JIHUSUOYOUDEBINGXINGSHUJUKUXITONGDOUYONGDAOLEZHEILIANGGEXIANGMUDEJISHU: GUANXIBIAODESHUIPINGFENQUYIJI SQL CHAXUNYUJUDEFENQUZHIXING[15].SHUIPINGFENQUDEBEIHOUSIXIANGSHIJIANGGUANXIBIAODEXINGZAIJIQUNJIEDIANSHANGJINXINGHUAFEN,ZHEIYANGTAMENJIUKEYIBEIBINGXINGDICHULI。BIRUJIANGYIGE 10-million-XINGDEBIAOZAIYIGE 50 JIEDIANDEJIQUNSHANGJINXINGHUAFEN,MEIGEJIEDIANJUYOU 4 GECIPAN,ZHEIYANGMEIGECIPANSHANGJIANGHUIYOU 50 000 GEXING。JUEDADUOSHUDEBINGXINGSHUJUKUXITONGDOUTIGONGLEDUOZHONGHUAFENCELVE,BAOKUO Hash,range YIJI round-robin HUAFEN[8].ZAIJIYU Hash DEFENQUMOSHIXIA,DANGMEIXINGBEIJIAZAIDESHIHOU,HUIDUIMEIXINGLIDEYIGEHUODUOGESHUXINGYINGYONGYIGEHash HANSHULAIJUEDINGGAIXINGSUOYINGGAICUNFANGDEMUBIAOJIEDIANHECIPAN。

  CONG 20 SHIJI 90 NIANDAIZHIJIN,SUIZHECHULIQI、CUNCHU、WANGLUODENGXIANGGUANJICHUJISHUDEFAZHAN,BINGXINGSHUJUKUJISHUDEYANJIUSHANGSHENGDAOYIGEXINDESHUIPING,YANJIUDEZHONGDIANYEZHUANYIDAOSHUJUCAOZUODESHIJIANBINGXINGXINGHEKONGJIANBINGXINGXINGSHANG。NENGGOUBINGXINGZHIXINGYOULIANGGEGUANJIAN: 1) DABUFEN( SHENZHISHISUOYOUDE) BIAOBEIHUAFENDAOJIQUNDESUOYOUJIEDIANSHANG; 2) XITONGSHIYONGYIGEYOUHUAQILAIJIANG SQL MINGLINGFANYICHENGZAIDUOGEJIEDIANSHANGZHIXINGDECHAXUNJIHUA。YINWEICHENGXUYUANZHIXUYAOSHIYONGGAOJIYUYANMIAOSHUTAMENDEMUDE,YINCITAMENGENBENBUXUYAOGUANXINDICENGCUNCHUXIJIE,BIRUSUOYINPEIZHIHELIANJIECELVE。BINGXINGSHUJUKUXITONGDEMUBIAOSHIGAOXINGNENGHEGAOKEYONGXING,TONGGUODUOGECHULIJIEDIANBINGXINGZHIXINGSHUJUKURENWU,TIGAOZHENGGESHUJUKUXITONGDEXINGNENGHEKEYONGXING。XINGNENGZHIBIAOGUANZHUDESHIBINGXINGSHUJUKUXITONGDECHULINENGLI,JUTIDEBIAOXIANKEYITONGYIZONGJIEWEISHUJUKUXITONGCHULISHIWUDEXIANGYINGSHIJIAN。BINGXINGSHUJUKUXITONGDEGAOXINGNENGKEYICONGLIANGGEFANGMIANLIJIE,YIGESHISUDUTISHENG; YIGESHIFANWEITISHENG。SUDUTISHENGSHIZHI,TONGGUOBINGXINGCHULI,KEYISHIYONGGENGSHAODESHIJIANWANCHENGLIANGYANGDUODESHUJUKUSHIWU。FANWEITISHENGSHIZHI,TONGGUOBINGXINGCHULI,ZAIXIANGTONGDECHULISHIJIANNEI,KEYIWANCHENGGENGDUODESHUJUKUSHIWU。BINGXINGSHUJUKUXITONGJIYUDUOCHULIJIEDIANDEWULIJIEGOU,JIANGSHUJUKUGUANLIJISHUYUBINGXINGCHULIJISHUYOUJIJIEHE,LAISHIXIANXITONGDEGAOXINGNENG。

  KEYONGXINGZHIBIAOGUANZHUDESHIBINGXINGSHUJUKUXITONGDEJIANZHUANGXING,YEJIUSHIDANGBINGXINGCHULIJIEDIANZHONGDEYIGEJIEDIANHUODUOGEJIEDIANBUFENSHIXIAOHUOWANQUANSHIXIAOSHI,ZHENGGEXITONGDUIWAICHIXUXIANGYINGDENENGLI。GAOKEYONGXINGKEYITONGSHIZAIYINGJIANHERUANJIANLIANGGEFANGMIANTIGONGBAOZHANG。ZAIYINGJIANFANGMIAN,TONGGUORONGYUDECHULIJIEDIAN、CUNCHUSHEBEI、WANGLUOLIANLUDENGYINGJIANCUOSHI,KEYIBAOZHENGDANGXITONGZHONGMOUJIEDIANBUFENHUOWANQUANSHIXIAOSHI,QITADEYINGJIANSHEBEIKEYIJIESHOUQICHULI,DUIWAITIGONGCHIXUFUWU。

赢发彩票代理   ZAIRUANJIANFANGMIAN,TONGGUOZHUANGTAIJIANKONGYUGENZONG、HUXIANGBEIFEN、RIZHIDENGJISHUSHOUDUAN,KEYIBAOZHENGDANGQIANXITONGZHONGMOUJIEDIANBUFENHUOWANQUANSHIXIAOSHI,YOUTASUOJINXINGDECHULIHUOYOUTASUOZHANGKONGDEZIYUANKEYIWUSUNSHIHUOJIBENWUSUNSHIDIZHUANYIDAOQITAJIEDIAN,BINGYOUQITAJIEDIANJIXUDUIWAITIGONGFUWU。

赢发彩票代理   1. 2 MIANXIANGDASHUJUCHULIDE MapReduce MOXINGYANJIU

赢发彩票代理   2004 NIAN Google YANJIUYUAN Jeffrey Dean HE Sanjey Ghemawat TONGGUODUIWANGYESHUJUCUNCHUHEBINGXINGFENXICHULIYANJIUHOU,ZAIWENXIAN[1]TICHU Ma-pReduce JISUANMOXINGBINGZAI ACM DENGDUOGEQIKANSHANGZHUANZAI[2,3].MapRe-duce JISUANMOXINGWEIDASHUJUFENXICHULIWENTITIGONGLEYIGEXINDEYOUXIAOJIEJUEFANGFAHETUJING。WENXIANZHONGZHICHU,“MapRedcue SHIYIZHONGBIANCHENGMOXING,YIJIYIGECHULIHESHENGCHENGDASHUJUJIDEXIANGGUANSHIXIAN。CHENGXUBEIXIECHENGHANSHUSHI,BINGQIEZIDONGBINGXINGZHIXINGZAIYIGEDAGUIMODETONGYONGJISUANJIJIQUNSHANG。ZHEIYUNXUMEIYOURENHEBINGXINGHEFENBUSHIXITONGBIANCHENGJINGYANDECHENGXUYUANQINGSONGDILIYONGDAGUIMOFENBUSHIXITONGDEZIYUAN。”MapReduce JISUANMOXING Map CAOZUOTONGGUOBASHURUSHUJUJINXINGFENQU,LIRU: FENWEI M KUAI,FENBUDAOBUTONGDEJIQISHANGBINGXINGZHIXING。Reduce CAOZUOSHITONGGUODUIZHONGJIANCHANSHENGDE key DEJIANZHIDUILAIJINXINGFENBUDE,ZHONGJIANCHANSHENGDE key KEYIGENJUMOUZHONGFENQUHANSHUJINXINGFENBU,FENBUCHENGWEI R KUAI。FENQU( R) DESHULIANGHEFENQUHANSHUDOUSHIYOUYONGHUZHIDINGDE。JUTILIUCHENGRUTU 2 SUOSHI。

  ( 1) YONGHUCHENGXUZHONGDE MapReduce HANSHUKUSHOUXIANBASHURUWENJIANFENCHENGM KUAI,MEIKUAIDAXIAOWEI 16M DAO 64M( KEYITONGGUOCANSHUJUEDING) .JIEZHEZAIcluster DE worker JIQISHANGZHIXINGCHULICHENGXU。

  ( 2) 这些分排的执行程序中有一个程序比较特别,它是主控程序 master.剩下的执行程序都是作为 master 分排工作的worker.总共有 M 个 map 任务和 R 个 Reduce 任务需要分排。master 选择空闲的 worker 并且分配这些 map 任务或者 Reduce任务。

赢发彩票代理     ( 3) 一个分配了 map 任务的 worker 读取并处理相关的输入小块。它处理输入的数据,并且将分析出的 key/value 对传递给用户定义的 map 函数。map 函数产生的中间结果 key/value对暂时缓冲到内存。

赢发彩票代理   ( 4) ZHEIXIEHUANCHONGDAONEICUNDEZHONGJIANJIEGUOJIANGBEIDINGSHISHUAXIEDAOBENDIYINGPAN,ZHEIXIESHUJUTONGGUOFENQUHANSHUFENCHENG R GEQU。ZHEIXIEZHONGJIANJIEGUOZAIBENDIYINGPANDEWEIZHIXINXIJIANGBEIFASONGHUI master,RANHOUZHEIGE master FUZEBAZHEIXIEWEIZHIXINXICHUANSONGJI Reduce DE worker.

  ( 5) 当 master 通知 Reduce 的 worker 关于中间 key/value 对的位置时,它调用 remote procedure 来从 map worker 的本地硬盘上读取缓冲的中间数据。当 Reduce 的 worker 读到了所有的中间数据,就使用中间 key 进行排序,这样可以使得相同 key 的值都在一起。因为有许多不同 key 的 map 都对应相同的 Reduce任务,所以,排序是必须的。如果中间结果集太大,那么就需要使用外排序。

赢发彩票代理     ( 6) Reduce worker 根据每一个唯一中间 key 来遍历所有的排序后的中间数据,并且把 key 和相关的中间结果值集合传递给用户定义的 Reduce 函数。Reduce 函数对于本 Reduce 区块输出到一个最终的输出文件。

赢发彩票代理   ( 7) DANGSUOYOUDE map RENWUHE Reduce RENWUDOUYIJINGWANCHENGDESHIHOU,master JIHUOYONGHUCHENGXU。ZAIZHEISHIHOU MapReduce FANHUIYONGHUCHENGXUDEDIAOYONGDIAN。

  TONGGUOSHANGMIAN 7 GEBUZHOU,JIUSHUNLIZHIXINGWAN 1 GEWANZHENGDE MapReduceJISUANRENWU,KEJIANBINGXINGJISUANJISHUNENGXIANZHETIGAOSHUJUCHULINENGLI。

  1. 3 NoSQL YUSHUJUKUJISHUDEDUIBI

  FEIGUANXIXINGSHUJUKU( NoSQL) YIJIANZHIDUICUNCHU,TADEJIEGOUBUGUDING,MEIYIGEYUANZUKEYIYOUBUYIYANGDEZIDUAN,MEIGEYUANZUKEYIGENJUXUYAOZENGJIAYIXIEZIJIDEJIANZHIDUI,ZHEIYANGJIUBUHUIJUXIANYUGUDINGDEJIEGOU,KEYIJIANSHAOYIXIESHIJIANHEKONGJIANDEKAIXIAO。Google DE BigTable[13]JIUSHIDIANXINGDE NoSQLSHIXIAN。LINGWAIJUYOUDAIBIAOXINGDEHAIYOU Apache Cassandra YOU Facebook ZAIHadoop[23]JICHUSHANGKAIFADEHUNHEXINGDEFEIGUANXIDESHUJUKU,LEISIYUGoogle DE BigTable,YONGYUCHUCUNTEBIEDADESHUJU。

赢发彩票代理   YIKAISHI,SHUJUKULINGYUZHUANJIADUI MapReduce JISUANMOXINGWEIDAIBIAODEFEIGUANXIXINGSHUJUKUCHIFOUDINGTAIDU。2009 NIAN 6 YUE,ZAI SIGMOD 09 SHANGKANDENGLE Andrew Pavlo,Daniel J. Abadi,David J. DeWitt HE MichaelStonebraker DENGDEZHEMINGSHUJUKUHEZUODEWENZHANG“A Comparision of Ap-proaches to Large-Scale Data Analysis”[20],FENBIEDUIBINGXINGSHUJUKUHEMapReduce LIANGZHONGMOXINGJINXINGLEMIAOSHU。CIWAI,DUIBILEZHEILIANGZHONGXITONGDEXINGNENGHEKAIFADEFUZADU。ZUIHOU,DINGYILEYOUDUOGEJISUANRENWUZUCHENGDECESHIJI,BINGZAIYIGE MR DEKAIYUANBANBENHELIANGGEBINGXING DBMS XITONGSHANGJINXINGLECESHI。ZHENDUIMEIGEJISUANRENWU,BINGQIEZAIYIGE 100 JIEDIANDEJIQUNSHANGJINXINGLEDUOGEBINGXINGDUSHANGDEXITONGXINGNENGCESHI。DECHUJIELUN: JINGUANBINGXINGDBMS DESHUJUJIAZAIGUOCHENGHEZHIXINGDIAOYOUSUOHUAFEIDESHIJIANBI MR XITONGYAOZHANG,DANSHIZHEIXIE DBMS XITONGSUOBIAOXIANCHUDEZHIXINGXINGNENGQUEBI MR XITONGHAOHENDUO,BINGZUIHOURENDING: MapReduce ZAIDASHUJUCHULISHANGBIBINGXINGSHUJUKUCHA。

赢发彩票代理   SUIZHE MapReduce JISHUXINGNENGDETISHENG、YINGYONGLINGYUDEKUOZHAN,GUANXISHUJUGUANLIJISHUHE MapReduce JISHUDEZHENGLUNYIZHICHIXU。YUSHIZAI2010 NIAN 1 YUEDE ACM TONGXUNZAZHITONGSHIXIANG MIT DE Stonebraker JIAOSHOUHEGoogle DE Jeff Dean HE Sanjay Ghemawat YANJIUYUANJINXINGYUEGAO。Dean YAN JIU YUAN ZAI“MapReduce: A Flexible Data ProcessingTool”[3]ZHONGZHICHU MapReduce SHIJINXINGDAGUIMOSHUJUFENXICHULIDELINGHUOERYOUXIAODEGONGJU。YUBINGXINGSHUJUKUXIANGBI,MapReduce DEYOUSHIBAOKUOCUNCHUXITONGWUGUANYIJIDAGUIMO jobs DEXILIDURONGCUOXING。MapReduce SHIYIGEYONGYUDAGUIMOSHUJUJIHESHENGCHENGHECHULIDEBIANCHENGMOXING。YONGHUMIAOSHUYIGEMap HANSHUHE Reduce HANSHU,Map HANSHUHUICHULIYIGE key / value DUILAISHENGCHENGYIXILIEDEZHONGJIAN key/value DUIJIHE,Reduce HANSHUHUIDUIJUYOUXIANGTONGZHONGJIAN key ZHIDEZHONGJIAN values JINXINGHEBING。ZAI 2003 NIAN Dean JIUJIYUMapReduce MOXINGGOUJIANLEYIGEXITONGYONGLAIJIANHUA Google.com SUOSHIYONGDEDAOPAISUOYINDEGOUJIAN。CONGNEISHIQI,ZAI Google YIJINGYOUCHAOGUO 10 000 GEDEBUTONGCHENGXUSHIYONGLE MapReduce,HANGAILEYONGYUDAGUIMOTUCHULI、WENBENCHULI、JIQIXUEXI、JIQIFANYIDENGGEFANGMIANDESUANFA。MapReduce DE Ha-doop KAIYUANSHIXIANYEYIBEI Google ZHIWAIDEHENDUOZUZHIGUANGFANSHIYONG。YUBINGXINGSHUJUKUXIANGBI,MapReduce JUYOUHENDUOXIANZHEDEYOUSHI。SHOUXIANYESHIZUIZHONGYAODE,TAWEIDAGUIMO jobs TIGONGLEXILIDUDERONGCUOXING; ZAIYIGEXUYAOYUNXINGJIGEXIAOSHIDERENWUZHONGJIANCHUXIANCUOWUSHI,BUXUYAOCONGTOUKAISHI。QICI,MapReduce DUIYUYIGEJUYOUDUOGECUNCHUXITONGDEYIGOUXITONGDESHUJUCHULIHEJIAZAIFEICHANGYOUBANGZHU。DISAN,MapReduce TIGONGLEYIGEKEYIZHIXINGNEIXIEBI SQL SUONENGZHIJIEZHICHIDEGENGFUZAHANSHUDEHAOKUANGJIA。Stonebraker JIAOSHOUZAI“MapReduce and parallel DBMSs friendsor foes?”[29]ZHONGZECONGZUICHUDEDUI MapReduce JISHUDECHEDIFOUDING,ZHUANWEIKENDING MapReduce DELIANGHAOKUOZHANXING,BINGQIEZHICHU,MapReduce FEICHANGSHIHEZUO ETL ZHEIYANGDEGONGZUO。MUQIAN,YUELAIYUEDUODESHUJUKUYANJIURENYUANZHUJIANYISHIDAO,MapReduce HEGUANXISHUJUKUKEYIHUXIANGXUEXI,BINGQIEZOUXIANGJICHENG。

赢发彩票代理   MapReduce KEYICONG RDBMS XUEXICHAXUNYOUHUA、MOSHIZHICHI、WAIWEIGONGJUZHICHIDENG,ER RDBMS KEYICONG MapReduce XUEXIDEDAOGAODUDEKUOZHANXINGHERONGCUOXING、KUAISUZHUANGZAI、YIYUSHIYONGDENGTEDIAN。

  1. 4 MapReduce YUSHUJUKUJISHUXIANGJIEHEYANJIU

  ZAIBINGXINGSHUJUKUYU MapReduce MOXINGXIANGJIEHEDELILUNYANJIUFANGMIAN,GUOWAIYIYELUDAXUEDE Daniel J. Abadi YANJIUYUANWEIDAIBIAO,QIYANJIUTUANDUIJINSANNIANZAI SIGMOD,VLDB SHANGFABIAOLEDUOPIANGUANYUZAISHUJUKULINGYUDELIECUNCHUDELUNWEN[8 -11],FENBIEZAI 2009 NIANHE 2011 NIANFABIAOZAI VLDB SHANGDEHadoopDB[18,34]YANJIUWEIDAIBIAO,ZAI Apache Hadoop XIANGMUJICHUSHANGTICHULE Hadapt YANJIU,TAXIAOCHUSHUJUGUDAO,ZAIYUNHUANJINGZHONGSHIYONGXIANYOUDE SQLGONGJU,ZUZHIFENXIDALIANGDE“DUOCENGJIEGOU”SHUJU。YELUDAXUEJISUANJIKEXUEXIYANJIUKAIFADE Hadapt DEJISHU,JIEJUELEYIXIEZAIDASHUJUFENXIXUQIUZHONGDEGUANJIANWENTI: SHUJUDEJIASUZENGZHANG,FEIJIEGOUHUASHUJUDEBAOZHA。Hadoop-DB TONGGUOCHUANTONGGUANXISHUJUFENXIFANGFAFENXIJIEGOUHEFEIJIEGOUHUASHUJU。

  2011 NIANFABIAOZAI SIGMOD SHANGDEWENXIAN[34]GENGXIANGXIDIJIESHAOLETONGGUOMapReduce KUANGJIAYUSHUJUKUXITONGDEJIEHE,ZAIFENBUBINGXINGZHIXINGHUANJINGXIA,SHEJI HadoopDB SHUJUYINQING,TIGAOSHUJUFENXIXINGNENG,ZHONGDIANFENXILELIANJIEHEJUJICAOZUO。JIEHE Hadoop ZUOYEDIAODUHEWANGLUOCHUANBOJISHU,CHENGGONGDIJIANGDANJIEDIANSHUJUKUXITONGZHUANHUANWEIKEKUOZHANBINGXINGSHUJUKUFENXIPINGTAI。

  2010 NIANDE ICDE HE SIGMOD YIJI 2009 NIANDE VLDB SHANGFABIAOLE Facebook YANJIUYUANGUANYULIYONG MapReduce SHIXIANBINGXINGSHUJUCANGKUDEYANJIUCHENGGUO: Hive[31 -33].TASHIYIGEJIYU Hadoop DEKAIYUANSHUJUCANGKUJIEJUEFANGAN。RUTU 6 SUOSHI,Hive ZHICHIDECHAXUNSHILEISI SQL FANGSHIDECHENSHUYUYAN: HiveQL .ZHEIZHONGCHAXUNBEIBIANYIJIN MapReduce DE job YONGYIHadoop DEZHIXING。ERQIE HiveQL YUNXUYONGHUZAICHAXUNZHONGTIANJIA“KEHUmap-Reduce JIAOBENCHAJIAN”.ZHEIZHONGYUYANBAOHANLEYIGE“ZHICHIBAOHANYUANYULEIXINGDEBIAODE”SHUJUJIEGOU,LIRU,LIEBIAO、ZHAN、JIHE、YINGSHE,YIJIQIANTAOZUHESHUJUJIEGOU。QIBEIHOUDE I/O KUKEKUOZHANYICHAXUNKEHUGESHIDESHUJU。

赢发彩票代理   Hive HAIBAOHANLEYIGEXITONGMULU - YUANCUNCHU( Metastore) -BAOHANLE“YOULIYUSHUJUWAJUE、CHAXUNYOUHUAYUCHAXUNBIANYIDE”GAIYAOMOSHIYUTONGJI。

  ZAI Facebook,Hive SHUJUCANGKUBAOHANLESHUWANZHANGBIAO,GONGCUNCHUCHAOGUO 700TBDESHUJU。GUANGFANDIYONGYUYONGHUDEBAOGAOHEJISHIFENXI。ZAI 2011 NIANDE ICDE SHANG,Facebook YANJIUYUANFABIAOLE RCFile YIZHONGJIYU MapReduce DESHUJUCANGKUDEKUAISUQIECUNCHUGAOXIAODESHUJUFANGZHIJIEGOU[38],WENZHANGTICHUYIGEDADESHUJUBUJUJIEGOU,CHENGWEI RCFile( MIANXIANGJILUDELIECUNCHUWENJIANXITONG) ,RUTU 3 SUOSHI。

  BINGZAI Hadoop PINGTAISHANGSHISHI。TONGGUOMIJIDESHIYAN,WENZHANGXIANSHIDERCFile YOUXIAOXINGMANZUSIGEYAOQIU: ( 1) SHUJUDAORUKUAI; ( 2) CHAXUNCHULIKUAI; ( 3) CUNCHUKONGJIANLIYONGLVGAOXIAO; ( 4) DUIYUGAODUDONGTAIDEGONGZUOLIANGFANGSHIYAOYOUQIANGDADESHIYINGLI。RCFile YUXINGCUNCHUBIJIAOLESHUJUDAORUSUDUHEGONGZUOLIANGDESHIYINGLI。RCFile ZAIBIAOSAOMIAOTONGGUOBIMIANMEIBIYAOLIEZHIDUQULAIYOUHUADUQU,ZAIDADUOSHUQINGKUANGXIAYOUYUQITAJIEGOU。RCFile SHIJIYULIECUNCHUDEYASUO,YINCI,YOUHENGAODEKONGJIANLIYONGLV。RCFile YISHI Facebook DESHUJUCANGKUXITONGMORENXUANXIANG,YEYIYINGYONGYUYOU Facebook HEYAHUKAIFADESHUJUFENXIXITONG Hive ZHONG。

  2011 NIAN SIGMOD SHANGFABIAOLEXINJIAPOGUOLIDAXUEHUANGMINGJUNJIAOSHOUHEZHEJIANGDAXUECHENCHUNJIAOSHOUDEJIEZHULIECUNCHUJISHUSHIXIAN MapReduce KUANGJIAXIAKEKUOZHANLIANJIECHULILUNWEN[35].SHEJILE Llama ZHEIGEZAI MapReduce KUANGJIAXIADELIECUNCHUDESHUJUGUANLIYUANXINGXITONG,ZAIDICENGSHIYONGYIGECHUANGXINDEWENJIANCUNCHUGESHI: CFiles,RUTU 4 SUOSHI。ZAI CFile,MEIGEKUAIBAOHANGUDINGSHULIANGDEJILU,CHENGWEI K ZHI。MEIGELUOJIKUAIDEDUOSHAO n BUTONG,YINWEIJILUDAXIAOKEBIANDE。KUAICUNCHUZAIHUANCHONGQU。HUANCHONGQUDEDAXIAOTONGCHANGWEI 1 MB.DANGHUANCHONGQUDAXIAOCHAOCHUYUZHIHUOHUANCHONGQUZHONGDEJILUSHUDADAO k GE ,HUANCHONGQUSHUAXINDAO DFS.MEIKUAIDEQISHIPIANYILIANGBEIJILUXIALAI。SHIYONGDAKUAI chunk DAIBIAOZAIWENJIANXITONGDEFENQUDANWEI。ZAI HDFS ZHONGDEMEIGEWENJIANSHIQIECHENGKUAI,MEIDAKUAI chunk ZAIBUTONGDESHUJUJIEDIANFUZHI。ZAI HDFS,MOREN chunk DAXIAOWEI 64 MB.chunk BAOHANDUOGEKUAI,YOUJILU k DEZHIHEMEIGEJILUDEDAXIAOERDING。

  WENXIANLIHAISHEJILEBINGFALIANJIE,RUTU 5 SUOSHI,JIANGJINKENENGDUODELIANJIECAOZUOFANGZAI MapReduce KUANGJIA map JIEDUAN,TONGGUO Llama XITONGGOUJIAN,SHIXIANLEDUIDASHUJUDEKUAISUCHAXUN,BINGFALIANJIECHUZHONGSHIJIANGJINKENENGDUODELIANJIECAOZUOFANGZAI MapReduce KUANGJIA map JIEDUAN。JIBENSIXIANGSHILIYONGNONGMIDECHAXUNJIHUASHUHEPAIXU PF ZU,JIEJUE MapReduce RENWUZHONGDEDUOBIAOLIANJIEWENTI,BIMIANZHONGPAIANGGUIDESHUJUFUZHIHEJIANSHAO MapReduce RENWUSHU。GAISUANFAKEYIJIANQINGZAI HDFS DE NameNode JIEDIANBAOCHIDALIANGDEZHONGJIANJIEGUODEYALI。CIWAI,SHEJILESHUJUWUHUAHEBINGFALIANJIECHENGBENMOXINGLAIFENXISHUJUFANGWENCHENGBEN,CONGERDUIMOXINGJINXINGXIJIEYOUHUA。

  2011 NIAN VLDB SHANGFABIAOLEWEISIKANGXINGMAIDIXUNDAXUEHE IBM YANJIUYUANLIANHEYANFADEJIYULIECUNCHUJISHUDE MapReduce KUANGJIALUNWEN[36],LIYONGLIECUNCHUJISHUDUI DREMEL[25]DEGAIJIN: 1) JIANGFUZALEIXINGZUOWEIYIGEDANDUDELIECUNCHU,ERBUSHIXIANG DREMEL,JIANGQIFENJIEDAOBUTONGDELIEZHONG; 2) XITONGKEYICHULIZAI DREMEL LIWUFACHULI map SHUJULEIXING; 3) ZHONGDIANSHIZAI Hadoop DE Java DEBEIJINGXIADUIXINGNENGGAISHAN。SHOUXIAN,JIESHAOLIECUNCHUGESHIJIANRONG Hadoop FUZHIHEDIAODUYUESHUJIZHI,RUTU 6 SUOSHI,ZHENGMINGLIECUNCHUGESHIZAISHIJIGONGZUOFUZAITIAOJIANXIANENGJIAKUAI MapReduce RENWUCHULISUDU;QICI、YANJIURUHECHULILIECUNCHUYUDAODEFUZADESHUJULEIXING,LIRU: SHUZULEIXING、MAP LEIXINGHEQIANTAOJILULEIXING。ZHEIXIEDOUSHI MapReduce LICHANGJIANZUIHOU,WENZHANGYINRUTIAOYUELIEBIAOLIECUNCHUGESHI,RUTU 7 SUOSHI,HE lazyJILUDEGOUJIANSUANFA,YIBIMIANBUXUYAODEJILUZUOFANXULIEHUASHIYANCAIYONGCONGIBM YANJIUZHONGXIN Intranet SHANGZHUAQUDEZHENSHISHUJU,SHIYANXIANSHILIECUNCHUJISHUKEYIZAI Hadoop Map JIEDUANSHIXIANGAODALIANGGESHULIANGJIXINGNENGTISHENG。

  2011 NIAN SIGMOD SHANGFABIAOLE Teradata YANJIUYUANHEJIAZHOUDAXUEHEZUODEJIYU Hadoop DEBINGXINGSHUJUCANGKUJIAZAIFANGFA[37]LUNWEN。SHIYONG HadoopLAIZUOWEIZHONGJIANJIAZAIFUWUQICUNCHUJIANGYAOJIAZAIDAO Teradata QIYEJISHUJUCANGKUDESHUJU。CONG HDFS( Hadoop FENBUSHIWENJIANXITONG) HUODELEZHUDUOYOUDIAN: 1) WEIYAOJIAZAIDEWENJIANDECIPANKONGJIANXIANZHEZENGJIA; 2) YIDANSHUJUBEIXIERUDAO HDFS,TASHIMEIYOUBIYAOBAOCUNSHUJUYUANDESHUJU,SHENZHIBEIJIAZAIDAOTeradata QIYEJISHUJUCANGKUZHIQIANDEWENJIAN; 3) MapReduce CHENGXUKEYONGYUGAIZAOHETIANJIAFEIJIEGOUHUAHUOBANJIEGOUHUASHUJUJIEGOU; 4) YINWEIWENJIANFENBUZAI HDFS ZHONG,SUOYIXITONGKEYIGENGXUNSUDIBINGXINGJIAZAIDAO Teradata QIYEJISHUJUCANGKUZHONG。DANG Hadoop HE Teradata QIYEJISHUJUCANGKUGONGCUNZAITONGYIYINGJIANSHANGPINGTAI,YOUYUJIANSHAOLEYINGJIANHEXITONGGUANLICHENGBEN,ZHENGYUELAIYUEDUODISHOUDAOKEHUDEQINGLAI; 5) LINGYIGEYOUHUAFANGFASHIJIANG HDFS SHUJUKUAIXIANGTONGDEJIEDIANSHANGZHIJIEJIAZAIDAO Teradata DEBINGXINGJIEDIANSHANG。YOUYU HDFS GUYOUDEFEIJUNYUNSHUJUFENBUTEXING,WENZHANGHENNANBIMIAN HDFS KUAIZHUANYIDAOPIANYUANDETeradata JIEDIAN。YINCI,WENZHANGSHEJILEYIGEDUOXIANGSHISHIJIANZUIYOUSUANFAHEDUOXIANGSHISHIJIANJINSIYOUHUASUANFA,HDFS KUAIJUNYUNFENPEIDAOBINGXINGDE TeradataJIEDIAN,BINGZUIDAXIANDUDIJIANSHAOLEWANGLUOLIULIANG。

  ZAIGUONEIDUIYUDASHUJUFENXIYINGYONGHE MapReduce YUSHUJUKUJISHUXIANGJIEHEJISHUYANJIU,XIANGDUIQIBUJIAOWAN。ZHONGGUORENMINDAXUEDETANXIONGPAIDENGFABIAOLE“DASHUJUFENXI---RDBMS YU MapReduce DEJINGZHENGYUGONGSHENG”[40]YIWEN,ZHICHUMIANDUIDASHUJUSHENDUFENXIDETIAOZHAN,GUANXISHUJUKUJISHUDEKUOZHANXINGYUDAOLEQIANSUOWEIYOUDEKUNNAN。TONGSHI,SQL DEBIAODANENGLIBUZUYIJINXINGFUZASHENRUDESHUJUFENXI。MapReduce JISHUJUYOUJIANJIEDEMOXING、LIANGHAODEKUOZHANXING、RONGCUOXINGHEBINGXINGXING,GAOXINGNENG。GUANXISHUJUKUJISHUHE Ma-pReduce JISHUXIANGHUJINGZHENG、XIANGHUXUEXIHEXIANGHUSHENTOU,CUJINLESHUJUFENXIXINSHENGTAIXITONGDEFUXIAN。ZAIXINSHENGTAIXITONGZHONG,GUANXISHUJUKUJISHUHE Ma-pReduce JISHUZHAODAOLEZIJIDEWEIZHI,FAHUICHUGEZIDEYOUSHI,CONGDASHUJUZHONGFENXIHEFAXIANYOUYONGDEZHISHI。GUANXISHUJUKUHE MapReduce JISHUGEYOUYOUQUEDIAN,RUHERONGHEGUANXISHUJUKUHE MapReduce JISHU,SHEJITONGSHIJUBEILIANGZHEYOUDIANDEJISHUJIAGOU,JIYOU MapReduce DEGAODUKUOZHANXINGHERONGCUOXING,YOUYOU RDBMS DEGAOXINGNENG,SHIDASHUJUFENXIJISHUDEYANJIUQUSHI。

赢发彩票代理   DONGBEIDAXUEYUGEDENG[41]TICHULEJIYU MapReduce DEGUANXIXINGSHUJUCANGKUBINGXINGCHAXUNFANGFA,BINGSHEJILEJIYU MapReduce DEFENBUSHIGUANXISHUJUKU: ChunkDB.NANJINGYOUDIANDAXUELILINGJUANDENG[42]TICHULEJIYU MapRe-duce DEPINFANXIANGJIWAJUEFANGFA,ZAISHUJUWAJUELIHUODEJIAOHAODESHIXIAOXING。

  WANGJINGDENG[43]ZAIHadoop JICHUSHANGTICHULEYIZHONGJIYUGUOLVQIDEDUOBIAOLIANJIESUANFA,JIANSHAOBUBIYAODEYUANZUFUZHIYUSHUJUCHUANSHU。ZAIGUONEIBINGXINGSHUJUKUYANJIUFANGMIAN,2000 NIAN,ZHONGGUORENMINDAXUEWENJIRONGDENGZAIWENXIAN[39]SHANGFENXILEBINGXINGSHUJUKUXITONGDECHAXUNYOUHUAJISHU。

赢发彩票代理   GAIWENJIESHAOZUOZHEZIXINGYANZHIDEYIGE Shared-nothing BINGXINGSHUJUKUXITONGPBASE /2 ZHONGDUTEDELIANGJIEDUANYOUHUACELVE。WEILESUOJIANBINGXINGCHAXUNYOUHUAPANGDADESOUSUOKONGJIAN,PBASE/2 JIANGBINGXINGCHAXUNYOUHUAHUAFENWEISHUNXUYOUHUAHEBINGXINGHUALIANGGEJIEDUAN。ZAISHUNXUYOUHUAJIEDUANDUIBINGXINGHUAHOUDETONGXINDAIJIAJINXINGYUXIANGUSUAN,JIANGTONGXINKAIXIAOJIARUSHUNXUYOUHUADEDAIJIAMOXING,TONGSHIDUIDONGTAIGUIHUASOUSUOSUANFAJINXINGLEXIUZHENGHEKUOZHAN,BAOZHENGLESHUNXUYOUHUAJIEDUANDEDAODEZUIXIAODAIJIAJIHUAZAIBINGXINGHUAHOUDAIJIARENGRANZUIXIAO。BINGXINGHUAJIEDUANDEYOUHUAMUBIAOSHISHIXIANCHAXUNGONGZUOLIANGZAIXITONGNEIDUOZHONGZIYUANSHANGDEFUZAIPINGHENG,TICHULEZIYUANFUZAIPINGHENGYINZIDEGAINIAN,BINGQIETONGGUOQIFASHIGUIZE、RENWUDIAODUDENGJIZHIBAOZHENGLEBINGXINGCHAXUNZHIXINGJIHUADEYOUHUAXING。

  2 JIE YU

  TONGGUOYISHANGFENXIKEYIKANCHU,DANGSHUJUJIHESUOYINBIANDASHI,CHUANTONGGUANXIXINGSHUJUKUZAIDUIDAGUIMOSHUJUJINXINGCAOZUOHUIZAOCHENGXITONGXINGNENGYANZHONGXIAJIANG,YINWEIZAICHULISHUJUSHI SQL QINGQIUHUIZHANYONGDALIANGDE CPU ZHOUQI,BINGQIEHUIDAOZHIDALIANGDECIPANDUXIE,XINGNENGHUIBIANMANDERANGRENWUFARENSHOU。

  SUIZHEDUI MapReduce BINGXINGJISUANJISHUYANJIUDESHENRUHE SMP、MPPDENGCHULIJIJISHUDEFAZHAN,MapReduce FENBUSHIBINGXINGCHULIJIQUNYIJINGCHENGWEILEDASHUJUYANJIUZHONGZUISHOUGUANZHUDEREDIAN。MUQIAN,MapReduce YUGUANXISHUJUKUJISHUXIANGJIEHELINGYUZHUYAOYOUXIALIEWENTIXUYAOJINYIBUDIYANJIUHEJIEJUE。

赢发彩票代理   ( 1) BINGXINGTIXIJIEGOUJIXITONGDEWULISHEJI

  WEILEDADAOBINGXINGCHULIDEMUDE,CANYUBINGXINGCHULIDEGEGECHULIJIEDIANZHIJIANSHIFOUYAOGONGXIANGZIYUAN、GONGXIANGNAXIEZIYUAN、XUYAODUODACHENGDUDEGONGXIANG,ZHEIXIEJIUXUYAOYANJIUBINGXINGCHULIDETIXIJIEGOUJIYOUGUANSHIXIANJISHU。ZHUYAOSHIZAIBINGXINGCHULIDEHUANJINGXIA,DASHUJUFENBUDESUANFADEYANJIU、SHUJUKUSHEJIGONGJUYUGUANLIGONGJUDEYANJIU。TEBIESHILIECUNCHUJISHUZAIBINGXINGHUANJINGXIASHIXIANDEYANJIU。

赢发彩票代理   ( 2) FEIGUANXIXINGSHUJUKU

  FEIGUANXIXINGSHUJUKUYIJIANZHIDUICUNCHU,TADEJIEGOUBUGUDING,MEIYIGEYUANZUKEYIYOUBUYIYANGDEZIDUAN,MEIGEYUANZUKEYIGENJUXUYAOZENGJIAYIXIEZIJIDEJIANZHIDUI,ZHEIYANGJIUSHIYONGDASHUJUZHONGDEBANJIEGOUHEFEIJIEGOUHUASHUJU,DANTADEJIEGOUBUBIANYUBIAOYUBIAOZHIJIANJINXINGLIANJIEDENGCAOZUO,ZHEISHIDANGQIANGUONEIWAICIXIANGYANJIUDENANDIANZHIYI。

赢发彩票代理   ( 3) CHULIJIEDIANJIANTONGXINJIZHIDEYANJIU

  WEILESHIXIANBINGXINGDEGAOXINGNENG,BINGXINGCHULIJIEDIANYAOZUIDACHENGDUDIXIETONGCHULISHUJUKUSHIWU,YINCI,JIEDIANJIANBIBUKESHAODICUNZAITONGXINWENTI,RUHEZHICHIDALIANGJIEDIANZHIJIANXIAOXIHESHUJUDEGAOXIAOTONGXIN,YECHENGWEILEBINGXINGSHUJUKUXITONGZHONGYIGEZHONGYAODEYANJIUKETI。

  ( 4) BINGXINGCAOZUOSUANFA

赢发彩票代理   WEITIGAOBINGXINGCHULIDEXIAOLV,XUYAOZAISHUJUFENBUBINGXINGDEYANJIUDEJICHUSHANG,SHENRUYANJIUCHUANTONGDELIANJIE、JUJI、TONGJI、PAIXUDENGJUTIDESHUJUCAOZUOZAI MapReduce JIEDIANSHANGDEBINGXINGCAOZUOSUANFA。ZHEISHIDANGQIANGUONEIWAICIXIANGYANJIUDEREDIANZHIYI。

赢发彩票代理   ( 5) BINGXINGCAOZUODECHAXUNYOUHUAWENTI

  WEIHUODEGAOXINGNENG,RUHEJIANGYIGESHUJUKUCHULIWULIZHIXINGHELIDIFENJIECHENGXIANGDUIDULIDEBINGXINGCAOZUOBUZHOU、RUHEJIANGZHEIXIEBUZHOUYIZUIYOUDEFANGSHIZAIDUOGECHULIJIEDIANJIANJINXINGFENPEI、RUHEZAIDUOGECHULIJIEDIANDETONGYIGEBUZHOUHEBUTONGBUZHOUZHIJIANJINXINGXIAOXIHESHUJUDETONGBU,ZHEIXIEWENTIDOUZHIDESHENRUYANJIU。

  ( 6) SHUJUDEJIAZAIHEZAIZUZHIJISHU

  WEILEBAOZHENGGAOXINGNENGHEGAOKEYONGXING,XITONGZHONGDECHULIJIEDIANKENENGXUYAOJINXINGKUOCHONG,ZHEIJIUXUYAOKAOLVRUHEJIANGCHUANTONG RDBMS DESHUJUJIAZAIFANGFAGAOXIAODIYIZHIDAO MapReduce KUANGJIAZHONGLAI,YIJIRUHEHELIDIZAIGEGEJIEDIANSHIZHONGXINZUZHISHUJU。

  ZONGSHANGSUOSHU,GUANXISHUJUKUHE MapReduce FEIGUANXIXINGSHUJUKUJISHUXIANGRONGHEYANJIUSHISHUJUKEXUE、SHUJUGONGCHENGLINGYUDEYANJIUQUSHI,WOGUOXIANGGUANYANJIUGONGZUOHAICHUYUQIBUJIEDUAN,GUOWAIDEYANJIUGONGZUOZHONGYEYOUHENDUOGUANJIANWENTIYOUDAIJIEJUE。DANGQIAN,GUONEIWAIGUANYUDASHUJU MapReduce KUANGJIAXIADECHULIYANJIUHAIZHUYAOSHIJIZHONGZAIKUANGJIAXIEYIDESHEJIFANGMIAN[5 -43],HAIMEIYOUYITAOWANZHENGDEJIEJUEFANGANHEXIANGGUANDEGUOJIBIAOZHUNXITONGKEYICAIYONG。

  CAN KAO WEN XIAN

赢发彩票代理   [1] Dean J,Ghemawat S. MapReduce: Simplified Data Processing on LargeClusters[C]/ / Proc of 6th OSDI. San Francisco: USENIX Association,2004: 137-150.

  [2] Dean J,Ghemawat S. Experiences with MapReduce: an abstraction forlarge scale computation[C]/ / Proc 15th Inter-Conf on PACT. Washing-ton DC,2006: 1-2.

  [3] Dean J,Ghemawat S. MapReduce: a flexible data processing tool[J].Communications of the ACM,2010,53: 72-77.

赢发彩票代理   [4] DeWitt D J,Gerber R H,Graefe G,et al. GAMMA-A High PerformanceDataflow Database Machine[C]/ / VLDB '86,1986: 228-237.

赢发彩票代理   [5] Fushimi S,Kitsuregawa M,Tanaka H. An Overview of The System Soft-ware of A Parallel Relational Database Machine[C]/ / VLDB '86,1986: 209-219.

对应分类:
版权所有:上海论文网专业权威的论文代写、论文发表的网站,秉承信誉至上、用户为首的服务理念,服务好每一位客户
本站部分论文收集于网络,如有不慎侵犯您的权益,请您及时致电或写信告知,我们将第一时间处理,邮箱:shlunwen@163.com