大數(shù)據(jù)存儲解決方案

從企業(yè)數(shù)據(jù)向大數(shù)據(jù)的擴展TraditionalApproachStructured,analytical,logicalSystemsofRecordNewApproach

Creative,holisticthought,intuitionSystemsOfEngagementMultimediaSystemsofInsight

EnterpriseIntegration

andContextAccumulationStructured

Repeatable

LinearUnstructured

Exploratory

DynamicDataWarehouseWebLogsSocialDataTextData:

emailsSensordata:

imagesRFIDInternalAppDataTransactionDataMainframeDataOLTPSystemDataHadoopand

StreamsTraditionalSourcesNewSourcesERP

data具備洞悉能力的系統(tǒng)SystemsofInsight對新式基礎(chǔ)架構(gòu)的需求在可靠和安全的環(huán)境中處理關(guān)鍵業(yè)務(wù)應(yīng)用存取和處理海量數(shù)據(jù)——包括結(jié)構(gòu)化和非結(jié)構(gòu)化數(shù)據(jù)速度及時響應(yīng)隨時可能出現(xiàn)的商業(yè)機會，這就需要靈活、實時性的基礎(chǔ)架構(gòu)ThedynamicsofSoRandSoE:通過負載及資源部署的優(yōu)化，來增強靈活性和效益通過采用包括基于開放標準的技術(shù)等新技術(shù)來改善ITeconomicsSystemofRecord（SoR）SystemsofEngagement（SoE）對的決策對的地方對的時間點BigData&Analytics大數(shù)據(jù)分析的新型架構(gòu)解決方案IBMBigData&AnalyticsInfrastructureDataZoneApplicationZone4SmartMeteringGridOperations電網(wǎng)管理FieldService外勤現(xiàn)場服務(wù)ResourcePlanning資源規(guī)劃CustomerService/CustomerOperations實現(xiàn)真正的有效的法規(guī)遵從及時發(fā)現(xiàn)能源損耗問題、以及偷電和欺詐行為提高客戶滿意度電量使用預(yù)測更為精確電網(wǎng)運維優(yōu)化減少停電次數(shù)和時間案例:SmartMetering智慧電力計費

大數(shù)據(jù)分析應(yīng)用可以帶來真正的業(yè)務(wù)價值法規(guī)遵從案例:用大數(shù)據(jù)分析來加強

SmartMetering數(shù)據(jù)分析的高可用性，以確保隨時了解用戶喜好跨應(yīng)用的TB級的數(shù)據(jù)需求–通用虛擬化存儲平臺實時收集、存儲并分析數(shù)據(jù)，最快可達50,000datapoints/sec歷史用電狀態(tài)數(shù)據(jù)的復(fù)雜查詢處理數(shù)據(jù)在加載到數(shù)據(jù)倉庫前的清洗、驗證，這些數(shù)據(jù)可能來自很多的用戶、收費系統(tǒng)或斷電保護系統(tǒng)關(guān)系掌控

構(gòu)建和維護電網(wǎng)的唯一試圖對整個企業(yè)的結(jié)構(gòu)化和非結(jié)構(gòu)化數(shù)據(jù)t做全局導(dǎo)覽Navigation，從中發(fā)現(xiàn)Discover價值分析用戶用電情況，偵測偷電、改表等行為預(yù)測哪些用戶適合于哪些分時時段電價或需求/響應(yīng)服務(wù)分時時段電價的實時定價或

提供及時的需求/響應(yīng)服務(wù)IBMBigData&AnalyticsReferenceArchitectureBigDataPlatformCapabilitiesInformationIngestReal-timeAnalyticsWarehouse&DataMartsAnalyticAppliancesAllDataSourcesAdvancedAnalytics/

NewInsightsNew/

EnhancedApplicationsCognitive認知LearnDynamically?Prescriptive規(guī)范BestOutcomes?Predictive預(yù)測WhatCouldHappen?Descriptive

描述WhatHasHappened?ExplorationandDiscoveryWhatDoYouHave?StreamingDataTextDataApplicationsDataTimeSeriesGeoSpatialRelationalSocialNetworkVideo&ImageAutomatedProcessCaseManagementAnalyticApplicationsWatsonCloudServicesISVSolutionsAlertsNewInfrastructureLeveragesDataTypesDatain

MotionDataat

RestDatain

ManyFormsInformationIngestionandOperationalInformationDecision

ManagementBIandPredictiveAnalyticsNavigation

andDiscoveryIntelligence

AnalysisRawDataStructuredDataTextAnalyticsDataMiningEntityAnalyticsMachineLearningLandingArea,AnalyticsZoneandArchiveVideo/AudioNetwork/SensorEntityAnalyticsPredictiveReal-timeAnalyticsExploration,IntegratedWarehouse,andMartZonesDiscoveryDeepReflectionOperationalPredictive

StreamProcessingDataIntegrationMasterDataStreamsInformationGovernance,SecurityandBusinessContinuityBigInsightsStreamsWarehouseInfoSphereBigInsightsHadoop-based低延遲分析，針對多樣化的、海量靜態(tài)數(shù)據(jù)Data-At-RestNetezzaHighCapacityAppliance基于結(jié)構(gòu)化數(shù)據(jù)的可查詢歸檔Netezza1000基于結(jié)構(gòu)化數(shù)據(jù)的

BI+定制化分析DataSmartAnalyticsSystem基于結(jié)構(gòu)化數(shù)據(jù)的運營分析InformixTimeseriesTime-structuredanalyticsInfoSphereWarehouse基于結(jié)構(gòu)化數(shù)據(jù)的大容量數(shù)據(jù)分析InfoSphereStreams低延遲流數(shù)據(jù)分析Velocity,Variety&VolumeData-In-MotionMPPDataWarehouseStreamComputingInformationIntegrationHadoopInfoSphereInformationServer海量數(shù)據(jù)集成和轉(zhuǎn)化ApacheHadoop:跨服務(wù)器集群的大數(shù)據(jù)集分布式處理開放系統(tǒng)框架，采用的是一種簡單化編程模型IBMBigDataPlatform大數(shù)據(jù)平臺What:一種開源軟件，將數(shù)據(jù)計算分布到整個集群的常見商用服務(wù)器和存儲上Why:傳統(tǒng)的計算架構(gòu)是一種沿縱向擴展模式，通過更快的SAN、大容量內(nèi)存和多級緩存將數(shù)據(jù)加載到CPU上，成本比較高。What:Hadoop把大數(shù)據(jù)集合拆分區(qū)劃為小數(shù)據(jù)集合，再把小數(shù)據(jù)集合分發(fā)到多臺普通服務(wù)器上，是一種橫向擴展模式。Why:Scalable,Flexible,CostEffective,FaultTolerentComponents:MapReduce,HDFSWhatisHadoop?NameNode(Metadatastore)NodesHDFSClusterOperatingSystemNodesElasticStorage-SNCClusterKernelLevelIBMValueforHadoop!HDFS把數(shù)據(jù)分散散存儲在多多個存儲節(jié)節(jié)點Node上HDFS設(shè)計時就假假設(shè)存儲節(jié)節(jié)點有失效效的可能，，所以HDFS會把一份數(shù)數(shù)據(jù)復(fù)制3份以上，分分散存儲在在多個節(jié)點點上，從而而實現(xiàn)系統(tǒng)統(tǒng)整體上的的可靠性HDFS文件系統(tǒng)是是由服務(wù)器器節(jié)點集群群組成的，，每臺服務(wù)務(wù)器依照HDFS的特有block協(xié)議支持網(wǎng)網(wǎng)絡(luò)化block數(shù)據(jù)HDFSNameNode有發(fā)生單點點故障的危危險IBM在改善文件件系統(tǒng)的性性能同時消消除了單點點故障——ElasticStorage-SNC(availableasbetacode)Hadoop說明,MapReduce,HDFSHadoopStackWhatdoesitlooklike?典型Hadoop存儲的PainPoints在選擇HDFS的組件（如如軟件、服服務(wù)器、網(wǎng)網(wǎng)絡(luò)和存儲儲等）時很很難選對在從測試環(huán)環(huán)境遷移到到生產(chǎn)環(huán)境境時，需要要做的調(diào)優(yōu)優(yōu)和調(diào)整工工作太繁復(fù)復(fù)了長期持續(xù)不不斷的運維維保障過于于繁重，比比如老要更更換失效組組件（尤其其是硬盤）），這使得得保證期望望的SLA非常難CPU和存儲去耦耦本來用戶的的CPU和內(nèi)存已經(jīng)經(jīng)滿足計算算需求，但但為了存儲儲容量需要要安裝更多多的硬盤不不得不買更更多的、不不必要的CPU和內(nèi)存Storageoptionsavailablehavecleargaps本地存儲的的利用率低低(~25%)，每次需要要擴容的時時候就要添添加更多的的服務(wù)器，，而一旦硬硬盤失效后后需要重建建，服務(wù)器器越多，失失效的幾率率越高，性性能也就越越差I(lǐng)BMStorageforHadoop傳統(tǒng)的Hadoop集群使用的的是服務(wù)器器內(nèi)置硬盤盤存儲。如如果用作測測試或科學(xué)學(xué)研究還好好，可作為為業(yè)務(wù)運行行的存儲就就要采用企企業(yè)存儲Hadoop集群要負責責數(shù)據(jù)保護護和復(fù)制重建（就是是copy）失效的數(shù)數(shù)據(jù)集到不不同節(jié)點上上——嚴重影響CPU性能，無法法實現(xiàn)企業(yè)業(yè)級的RASReplicatedata–問題同上擴展的時候候同時增加加處理器/網(wǎng)絡(luò)/存儲，無法法做到物盡盡其用（nowaytoseparatethese3evenifexcesscapacityexistinginone(e.g.NeededmorestoragebuthadtoaddComputeandNetwork)）使用外部存儲儲可以將存儲儲負載和Hadoop計算節(jié)點分離離，同時還獲獲得了企業(yè)存存儲的好處。。SellthevalueofXIV,V7000,SVC,etc.用戶一般會隨隨HadoopFileSystem部署；采用ElasticStorage可以有很多好好處14數(shù)據(jù)加速ExperiencetheinstantresultsthatcomefromIBMFlashSystemDriveasmuchas45Xfasteranalyticsresultsoncertainworkloads數(shù)據(jù)負載的多多樣性和靈活活性XIVdeliverspredictableperformancethatscaleslinearlywithouthotspotsdeliveringinsightsfromanalyticsfasterwithtuning-freedatadistributionScale-out,parallelprocessingofElasticStoragesoftwareandintegrationwithFlashSystemdramaticallyacceleratesperformanceofAnalyticsclustersVirtualStorageCenterwithSVCautomaticallyoptimizesdatawarehouseperformanceandcostacrossFlashandDiskMainframeDataEnvironmentsIntegrationwithDB2&specialtyanalytics“engines”leveragingDS8870delivers4xreductioninbatchtimeswithnewHighPerformanceFlashEnclosuresHighspeedencryptiononeverydrivetypesecuresdata數(shù)據(jù)保護和保保留LTFSEEw/tapeprovidesreducedTCObyupto90%overdiskforlongtermretentionofdataatrestwithalargeopenformattaperepositoryReducetheamountofdatatobestoredbyupto25timeswithProtecTIERde-duplication12x更快IBMFlashSystemincreasedSPLUNK&SASapplicationefficiencytoperformbusinessanalytics20x改善inactionablesupplychainanalytics,4xreductioninbatchtimes,virtualizationforplug&play6x時間節(jié)省“GPFSallowsustomovethemetadatafromthedisktotheFlashSystemonline.Oncewedidthat,thebackupswerereduceddowntoaboutanhour.”2hrsbecomes2minutes失效切換時間間大幅縮短MappingCharacteristicstoIBMStorageProductsStorageInfrastructure需求適用于所有的的5種應(yīng)用場景OptimizedMulti-TemperatureWarehouse優(yōu)化的多級存存儲庫AllFlashFlashSystemHybridDS8000EasyTierXIV+SSDCachingStorwizeEasyTierFlashSystemSolution(VSC+FlashSystem)PureSystemsPureFlex(XIVorStorwizew/EasyTier)PureDataforTransactions(Storwize)PureDataforAnalytics(Netezza)Midrange&EntryTier0AccelerationSmarterStorageIntegratedSystemsEnterpriseOfferingsXIVzEnterpriseSolutionsforAnalyticswithDS8000PureDataSystemforOperationalAnalyticswithStorwizePureFlexSystemwithStorwizeDS8000SmartAnalyticsSystemswithDS3xxxOpen&ExtensibleStorwizefamilyFlashSystemfamilyIBMSmarterStorage的設(shè)計就是支支持大數(shù)據(jù)分分析高效和優(yōu)化數(shù)數(shù)據(jù)基礎(chǔ)架構(gòu)構(gòu)IBMFlashSystem：為大數(shù)據(jù)分分析應(yīng)用設(shè)計計的，讓應(yīng)用用和數(shù)據(jù)實現(xiàn)現(xiàn)極速IBMFlashSystem的極速性能讓實時業(yè)務(wù)決決策成為可能能適合于模塊化化數(shù)據(jù)存儲結(jié)結(jié)構(gòu)的Hadoop系統(tǒng)。某些或或所有數(shù)據(jù)可可以保存到Flash閃存上，其他他可以保存到到XIVIBMXIV:OptimizeddataworkloaddiversityforBigData&AnalyticsIBMXIV的高性能無須人工干預(yù)預(yù)配置，且適適用于各種各各樣的存儲負負載IBMXIV的效率高的異乎尋常常，而且簡單單性業(yè)內(nèi)最高高，內(nèi)置友好好界面IBMXIV的彈性是企業(yè)級的，，完全保證了了數(shù)據(jù)的可用用性和業(yè)務(wù)連連續(xù)性XIV:為Analytics而生無與倫比的性能可擴展的網(wǎng)格格存儲架構(gòu)任意時間支持持任意讀寫負負載板上的閃存Flash無與倫比的可靠性精致的數(shù)據(jù)分分布無雙的磁盤重重建時間企業(yè)級的可用用性無與倫比的簡易性簡單的規(guī)劃、、供給和靈活活性上線后零維護護零調(diào)優(yōu)“XIV最吸引我們的的地方就是其其超強的性能能…we正是由于XIV為我們的精細細復(fù)雜的分析析應(yīng)用提供了了一致的高性性能，使得得我們能夠為為我們的用戶戶帶來更多的的價值?！盨AS和XIV網(wǎng)格架架構(gòu)——完美的的結(jié)合合大規(guī)模模并行行計算算保持持持續(xù)地地最佳佳性能能BalancedPerformance性能均均衡常年零零調(diào)整整UnprecedentedScalability史無前前例的的擴展展性配合添添加SAS節(jié)點和和XIV模塊即即可IBMSVC:OptimizeddataworkloadflexibilityforBigData&AnalyticsIBMSVC通過如如下功功能在在IBM大數(shù)據(jù)據(jù)產(chǎn)品品線上上增加加了靈活性性：完整和和數(shù)據(jù)據(jù)虛擬擬化和和數(shù)據(jù)據(jù)移動動性高級集集群和和復(fù)制制多路鏡鏡像，，readpreferredoptionRealTimeCompression實時壓壓縮EasyTierHotExtentcachingStorwizeV7000/UIBMSVC設(shè)計原原則Real-TimeCompression實時壓壓縮是是設(shè)計計來做做：作用于于ActivePrimaryData專用的的壓縮縮平臺臺PlatformhandlesALLheavyliftingassociatedwithcompression不會影影響性性能Wemodifyacompressedfilein-placeefficiently不會改改變用用戶應(yīng)應(yīng)用Usersnoradminsneedtochangeanything處理流流程不不變壓縮是是在線線完成成，不不是事事后壓壓縮業(yè)界標標準壓壓縮算算法所采用用的壓壓縮算算法已已經(jīng)使使用了了幾十十年StorwizeV7000/UIBMSVC24流處理理計算算&IBMFlashSystemsData:是擁有有還是是保存存?或是是是分析析和開開始行行動!DatainDataat25InfoSphereStreams:大數(shù)據(jù)據(jù)流分分析為分析析動態(tài)態(tài)數(shù)據(jù)據(jù)而建建多并發(fā)發(fā)輸入入數(shù)據(jù)據(jù)流大規(guī)模?？蓴U擴展Massivescalability分析和和處理理的數(shù)數(shù)據(jù)多多樣化化Structured,unstructured,video,audioAdvancedanalyticoperators自適應(yīng)應(yīng)實時時分析析WithDataWarehousesWithHadoopSystemsCurrentfactfinding當前數(shù)數(shù)據(jù)查查詢分許流流動中中的數(shù)數(shù)據(jù)——在數(shù)據(jù)據(jù)落盤盤前低延遲遲模式式,pushmodel數(shù)據(jù)驅(qū)驅(qū)動——真正的的數(shù)據(jù)據(jù)分析析Historicalfactfinding歷史數(shù)數(shù)據(jù)查查詢查找和和分析析存儲儲在磁磁盤上上的數(shù)數(shù)據(jù)信信息批處理理模式式,pullmodel查詢驅(qū)驅(qū)動:submitsqueriestostaticdataTraditionalComputingStreamComputing流數(shù)據(jù)據(jù)計算算代表表著計計算模模式的的變遷遷Real-timeAnalyticsRealTimeAnalytics實時分分析想象一一下你你如何何用防防火栓栓喝水水來自多多個多多樣輸輸入源源的大大量數(shù)數(shù)據(jù)直接處處理和和過濾濾數(shù)據(jù)據(jù)，而而不必必存儲儲僅保存存有價價值的的數(shù)據(jù)據(jù)僅關(guān)聯(lián)聯(lián)對數(shù)數(shù)據(jù)最最感興興趣的的用戶戶隨著數(shù)數(shù)據(jù)信信息的的產(chǎn)生生采取取行動動AdaptiveAnalytics自適應(yīng)應(yīng)分析析DatainMotionandDataatRest的集成成1.DataIngest數(shù)據(jù)集集成，，數(shù)據(jù)挖挖掘，，機器學(xué)學(xué)習，，統(tǒng)計建建模實時和和歷史史數(shù)據(jù)據(jù)洞察察力的的可視視化3.AdaptiveAnalyticsModel數(shù)據(jù)收收取，，在線分分析準準備，，模式式校驗驗Data2.Bootstrap/EnrichControlflowInfoSphereBigInsights,Database&WarehouseInfoSphereStreamsAdaptiveReal-TimeAnalytics自適適應(yīng)應(yīng)實實時時分分析析來自自多多個個多多樣樣輸輸入入源源的的大大量量數(shù)數(shù)據(jù)據(jù)過去去、、現(xiàn)現(xiàn)在在和和未未來來全全方方位位綜綜合合性性視視圖圖實時時分分析析，，低低延延時時結(jié)結(jié)果果Fullcontextfordeepanalysis深度度分分析析的的完完整整的的上上下下文文跨datainmotionanddataatrest的常常用用數(shù)數(shù)據(jù)據(jù)分分析析自適適應(yīng)應(yīng)-隨機機而而變變當發(fā)發(fā)現(xiàn)現(xiàn)非非預(yù)預(yù)期期行行為為時時，，自自適適應(yīng)應(yīng)當識識別別出出新新數(shù)數(shù)據(jù)據(jù)意意義義時時深深度度分分析析之之開始始沒沒有有意意識識到到的的數(shù)數(shù)據(jù)據(jù)意意義義，，隨隨后后才才可可能能意意識識到到自適適應(yīng)應(yīng)———在開開始始沒沒有有意意識識到到的的，，隨隨后后可可以以找找出出數(shù)數(shù)據(jù)據(jù)模模式式StockmarketImpactofweatheronsecuritiespricesAnalyzemarketdataatultra-lowlatenciesMomentumCalculatorFraudpreventionDetectingmulti-partyfraudRealtimefraudpreventione-ScienceSpaceweatherpredictionDetectionoftransienteventsSynchrotronatomicresearchGenomicResearchTransportationIntelligenttrafficmanagementAutomotiveTelematicsEnergy&UtilitiesTransactivecontrolPhasorMonitoringUnitDownholesensormonitoringNaturalSystemsWildfiremanagementWatermanagementOtherManufacturingTextAnalysisERPforCommoditiesReal-timemultimodalsurveillanceSituationalawarenessCybersecuritydetectionLawEnforcement,

Defense&CyberSecurityHealth&LifeSciencesICUmonitoringEpidemicearlywarningsystemRemotehealthcaremonitoringTelephonyCDRprocessingSocialanalysisChurnpredictionGeomapping如何使用用InfoSphereStreams?加快數(shù)據(jù)據(jù)流入分分析系統(tǒng)統(tǒng)的速度度向交易方方向加速速。。。。一個高效效和靈活活的基礎(chǔ)礎(chǔ)架構(gòu)顯顯然可以以加快流流速，并并平衡不不同數(shù)據(jù)據(jù)分析的的需求CoresSCMStorageNetworkCoresSCMStorageNetworkCoresSCMStorageNetworkCoresSCMStorageNetwork++預(yù)測分析

數(shù)據(jù)倉庫文本分析HadoopWorkloads優(yōu)化敏感性分析加快流速價值時間“觸發(fā)事件”數(shù)據(jù)完備交易Insight預(yù)見獲取數(shù)據(jù)時間分析數(shù)據(jù)時間行動時間大數(shù)據(jù)分分析的新新式基礎(chǔ)礎(chǔ)架構(gòu)解解決方案案IBMBigData&AnalyticsInfrastructureDataZoneApplicationZoneExperiencereal-timeanalyticalinsightswithupto50xbetterperformancethanenterprisedisksystemsusingIBMFlashCore??technologyPreserveandprotectinfrastructurecontinuitywhilescalingtoover2petabyteofeffectiveall-flashcapacityunderasingleintegrateinterfaceDeliveragilityanddataeconomicswith4xgreatercapacityinlessrackspacethancompetitiveall-flashproductsSynchronizedandComplimentarytoOverarchingStorageMessaging-Acceleratetimetoinsightsthrough"datawithoutborders."IBMinnovationfreesdatawithagileandsimpletousestoragesolutionsdeliveringsuperiordataeconomicsIBMFlashSystemCoreLaunchMessagingDriveacompleteparadigmshiftinEnterpriseStoragewiththeallnewIBMFlashSystemFamilyIBMFlashSystemFamily2015ThemeTimetoinsight.Timetovalue.Timetomarket.IBMFlashSystem,it’sabouttime.FlashRealized!IBMFlashSystemV9000FoundationalPillarsIBMFlashCore?TechnologyistheDNAoftheFlashSystemFamilyScalablePerformanceEnduringEconomicsAgileIntegrationIntroducingtheNewIBMFlashSystemFamilyOfferingsIBMFlashSystem900ExtremePerformance:Delivers100microsecondresponsetimesMacroEfficiency:Lowestlatencyofferingwith>40%greatercapacityatalowercostpercapacityEnterpriseReliability:IBMenhancedMicronMLCflashtechnologywithFlashWearGuaranteePoweredbyIBMFlashCore?TechnologyIBMFlashSystemV9000ScalablePerformance:Growcapacityandperformancewithupto2.2PBscalingcapabilityEnduringEconomics:NextgenerationflashmediawithlowercostpercapacityAgileIntegration:FullyintegratedsystemmanagementtosimplifymanagementandimproveworkforceproductivityunderasinglenamespaceFlashSystem900IntroducingIBMFlashSystem900,thenextgenerationinourlowes