HugeGraph圖數(shù)據(jù)庫(kù)平臺(tái)介紹

1、HugeGraph圖數(shù)據(jù)庫(kù)平臺(tái)介紹技術(shù)創(chuàng)新，變革未來目錄圖數(shù)據(jù)庫(kù)是什么HugeGraph是什么HugeGraph典型案例HugeGraph存儲(chǔ)原理如何參與開源貢獻(xiàn)3圖數(shù)據(jù)庫(kù)是什么新型的NoSQL數(shù)據(jù)庫(kù)4Key-ValueDocumentColumnGraphRedis RocksDBHBase CassandraCouchDB MongoDBNeo4j TitanDB查詢復(fù)雜度數(shù) 據(jù) 關(guān) 聯(lián) 度圖數(shù)據(jù)庫(kù)是什么圖數(shù)據(jù)庫(kù)是一種新型的NoSQL數(shù)據(jù)庫(kù)5圖數(shù)據(jù)庫(kù)是什么圖數(shù)據(jù)庫(kù)在近幾年廣受關(guān)注6圖數(shù)據(jù)庫(kù)是什么圖數(shù)據(jù)庫(kù)：存儲(chǔ)實(shí)體與實(shí)體之間的關(guān)聯(lián)關(guān)系G = (V, E)Vertex (Node, EnIty

2、, Object)Edge (RelaIonship, Link, Arc)現(xiàn)實(shí)世界圖數(shù)據(jù)庫(kù)7圖數(shù)據(jù)庫(kù)是什么8idnameagephone1Tom22188*11112Mike23158*22221. useriduser1user2since1122016-01-0222. friend好友關(guān)系用 用 戶戶圖數(shù)據(jù)庫(kù)帶來的改變SQL圖數(shù)據(jù)庫(kù)select * from friend ajoin friend b on b.user1=a.user2 join user on id=b.user2where a.user1 in (select id from user where name=To

3、m)Tomg.V().has(name, Tom).out(friend).out(friend)9圖數(shù)據(jù)庫(kù)帶來的改變10數(shù)據(jù)來自程序員深度關(guān)系型數(shù)據(jù)庫(kù)執(zhí)行時(shí)間 / s圖數(shù)據(jù)庫(kù)執(zhí)行時(shí)間 / s返回結(jié)果數(shù)量20.0160.012500330.270.16811000041543.511.35960000052.132800000圖數(shù)據(jù)庫(kù)應(yīng)用行業(yè)反欺詐發(fā)現(xiàn)人員、事件、地點(diǎn) 和時(shí)間之間的異常聯(lián)系知識(shí)圖譜構(gòu)建實(shí)體與實(shí)體間的關(guān) 聯(lián)關(guān)系，描述真實(shí)世界網(wǎng)絡(luò)安全服務(wù)器、域名、IP、文件 等關(guān)聯(lián)關(guān)系，形成網(wǎng)絡(luò)安 全情報(bào)IT運(yùn)維系統(tǒng)調(diào)用，數(shù)據(jù)庫(kù)訪問， 緩存讀取等全鏈路監(jiān)控更多圖數(shù)據(jù)庫(kù)應(yīng)用金融社交招聘物流醫(yī)療電信零

4、售圖數(shù)據(jù)庫(kù)11HugeGraph是什么百度安全自研的開源圖數(shù)據(jù)庫(kù)12HugeGraph特點(diǎn)HugeGraph易用高效通用開放可擴(kuò)展大規(guī)模1314PersonRobert ZemeckisPersonNora EphronHugeGraph優(yōu)勢(shì)MovieSleepless in SealeMovieYouve Got MaildirecteddirectedMovieWhen Harry Met SallywrotePersonMeg RyanactedPersonactedTom HanksactedMovieCast AwayactedactedMovieactedJoe Versus Vo

5、lcanoactedacteddirectedMovieThe Polar Express directed關(guān)系數(shù)據(jù)庫(kù) MySQL 百萬千萬規(guī)模十億+規(guī)模圖數(shù)據(jù)庫(kù) HugeGraph易 建 模大 規(guī) 模782345likelikefriendfriendfriendfriendlikelikelike?6like9like1personmoviev1v2v3v4v5v2v6v7v3v8v4v8v5v8v9高 效 關(guān) 聯(lián) 分 析/ Gremlin 2層好友查詢g.V().has(name, Tom Hanks).out(friend).out(friend)2362345friend1Tom H

6、anksfriendfriend4friend5 friendfriend71source6target靈 活 查 詢語 / 查詢2點(diǎn)之間的最短路徑g.V(source_id)言 .repeat(out().simplePath().until(hasId(target_id).path().limit(1)HugeGraph技術(shù)選型RocksDBCassandraHBase4. 持久化方案Non-Na3ve StorageNaIve Storage3. 查詢語言GremlinCypherSPARQLg.V().has(name,gremlin).out(knows).out(knows).v

7、alues(name)2. 存儲(chǔ)模型A: (B,D)B: (C,D)C: (E)D: (B,C,E)E: (A,C)F: (D,E)Adjacency ListAdjacency Matrix1. 概念模型Property GraphRDFpersonpersonpersonpersonsoftsoft15HugeGraph整體架構(gòu)OLAPMonitorStudioLoaderToolsApplicaIonsClientGraph EngineBackend Storage InterfaceJob Manager & ExecutorOLTPMonitor APIReskul APIGrem

8、lin API Internal API Layer (Schema, Graph, Index)Gremlin Job APIRocksDBCassandraHBasePalo | TableInMemoryScyllaDBMySQL(Ext Plugins)16HugeGraph性能批量寫入性能最短路徑查詢性能數(shù)據(jù)集后端email-enron (30w edge)amazon0601 (300w edge)com-youtube.ungraph (300w edge)com-lj.ungraph (3000w edge)HugeGraph1.72613.06613.009141.212Ti

9、tan14.02125.975154.9261467.159Neo4J4.69419.39622.199447.488數(shù)據(jù)集后端email-enron (30w edge)Amazon0601 (300w edge)com-youtube.ungraph (300w edge)com-lj.ungraph (3000w edge)HugeGraph2.2890.24210.21828.78Titan13.3260.577544.492848.36Neo4J2.0013.8995.93728.9251710億邊導(dǎo)入 BSKU, KGProduct-BSKU,KGUser, Doc - BSKU,

10、 KGKG-libquery-kgproduct-kgAD-kguser-kgdoc-kg結(jié)構(gòu)化解析實(shí)體鏈接 / 屬性填充Score / Rank知識(shí)沉淀 圖譜進(jìn)化HugeGraph知識(shí)沉淀 / 圖譜進(jìn)化實(shí)體挖掘 圖譜構(gòu)建實(shí)體識(shí)別 / 專名識(shí)別 屬性識(shí)別 / 屬性填充 實(shí) 體 挖 掘關(guān)系挖掘 / 共引消歧 實(shí)體融合 / 置信計(jì)算 圖 譜 構(gòu) 建數(shù)據(jù)處理數(shù)據(jù)引入 / 數(shù)據(jù)清洗數(shù)據(jù)解析 / 結(jié)構(gòu)提取實(shí)體和實(shí)體關(guān)系蘊(yùn)含規(guī)則和知識(shí)典型圖數(shù)據(jù)應(yīng)用場(chǎng)景20基于知識(shí)圖譜的廣告推薦HugeGraph圖遍歷切詞切詞 組合查找 搜索詞追溯 行業(yè)路徑結(jié)果 打分 根據(jù)屬性 路徑 展示排序過濾商品融合1、商品: Len

11、ovo拯救者Y7000.2、商品: Lenovo330C.3、商品: Lenovo K5.21金融風(fēng)控ID1ID3ID2ID唯一性檢查循環(huán)擔(dān)保檢測(cè)二度關(guān)系觸黑黑用戶風(fēng)險(xiǎn)評(píng)分?jǐn)U散1. 每個(gè)頂點(diǎn)一個(gè)組2. 將頂點(diǎn)加入鄰居 標(biāo)簽數(shù)目最多的組3. 組信息不再變化LPA 算法Louvain 算法屬性特征檢測(cè)關(guān)系特征檢測(cè)團(tuán)伙檢測(cè)22HDFSHivePaloHDFSHiveKafkaMySQLColumn PartitionIT資產(chǎn)IDC百度云HadoopAFS業(yè)務(wù)組織移動(dòng)安全AI安全安全SDK手機(jī)衛(wèi)士人員張三KafkaHiveHDFSETLEdge Vertex 大數(shù)據(jù)安全治理數(shù)據(jù)資產(chǎn)資產(chǎn) 視 圖數(shù) 據(jù)

12、血 緣數(shù)據(jù)安全隱私保護(hù)資產(chǎn)管理數(shù)據(jù)開發(fā)HugeGraph李四23王二HugeGraph存儲(chǔ)原理存儲(chǔ)結(jié)構(gòu)：邊集數(shù)組、鄰接矩陣、鄰接表、十字鏈表HugeGraph存儲(chǔ)結(jié)構(gòu)：鄰接表（順序表）24圖存儲(chǔ)結(jié)構(gòu)CSVHDFSJDBCGraph25Node邊集數(shù)組26邊ID源頂點(diǎn)目標(biāo)頂點(diǎn)邊屬性e1v1v2weighte2v1v3weighte3v2v6scoreae4v2v7scoree5v1v4weighte6v1v5weighte7v2v3weighte8v5v4weight172345e16 e2e5e6e3e4e7e8鄰接矩陣27v1v2v3v4v5v6v7v10123400v20070056v30

13、000000v40000900v50008000v60000000v7000000017234516 23456789鄰接表2817234516234567891110v1v2v3v5v2v6v7v3v3v4v4v5v5v4v6v7v6v1頂點(diǎn)下條邊十字鏈表2917234516 2345678912v113142627v2v3v4v5v615v72354 45鄰接表（順序表）30v3v4v5v6v7v7v2v1v2v5v5v4v1v417234516234567891110v6v3v1v2v4v1v5v2v4v3v2v6v1v3 v7出邊入邊如何參與開源貢獻(xiàn)Issue 問題與需求反饋Pull Request 貢獻(xiàn)代碼與評(píng)審31問題與需求反饋hps:/hugegraph/hugegraph/issues32貢獻(xiàn)代碼與評(píng)審1 fork2 git clone6 git push7 create prReview9 approveCode8 reviewrequest changes3 git checkout -b4 x c