數(shù)據(jù)挖掘概論.課件

1、What is Data Mining ?數(shù)據(jù)挖掘概論南京航空航天大學信息科學與技術學院皮德常 教授、博導Lots of data is being collected and warehoused Web data, e-commercepurchases at department/grocery storesBank/Credit Card transactionsComputers have become cheaper and more powerfulCompetitive pressure is strong Provide better, customized services

2、 for an edge (e.g. in Customer Relationship Management)Why Mine Data? Commercial ViewpointWhy Mine Data? Scientific ViewpointData collected and stored at enormous speeds (GB/hour)remote sensors on a satellitetelescopes scanning the skiesmicroarrays generating gene expression datascientific simulatio

3、ns generating terabytes of dataTraditional techniques infeasible for raw dataData mining may help scientists in classifying and segmenting data, Mining Large Data Sets - Motivationdata rich but information poor!we are drowning in data, but starving for knowledge!哇！這么多的數(shù)據(jù)！怎樣才能用呢？挖！“Necessity is the m

4、other of invention”Data miningAutomated analysis of massive data setsMining Large Data Sets - MotivationA famous story:跟尿布一起購買最多的商品是啤酒！diapersbeerThe success of GoogleSearch Engine: Analyzing data on the internet to find what meets your demand.Larry Page 1973.3.26 & Sergey Brin 1973.8.21 166億美元 & 14

5、1億美元的財產(chǎn)，共享一架波音767 What is Data Mining?Data mining is the non-trivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns from huge volume of data. U. Fayyad, et al. s definition of KDD at KDD96What is (not) Data Mining? What is Data Mining? Certain names a

6、re more prevalent in certain US locations (OBrien, ORurke, OReilly in Boston area) What is not Data Mining? Look up phone number in phone directory Draws ideas from machine learning/AI, pattern recognition, statistics, and database systemsTraditional Techniquesmay be unsuitable due to Enormity of da

7、taHigh dimensionality of dataHeterogeneous, distributed nature of dataOrigins of Data MiningMachine Learning/Pattern RecognitionStatistics/AIData MiningDatabase systemsArchitecture: Typical Data Mining Systemdata cleaning, integration, and selectionDatabase or Data Warehouse ServerData Mining Engine

8、Pattern EvaluationGraphical User InterfaceKnowle-dgeBaseDBDWWWWOther InfoRepositoriesData Mining TasksPredictionUse some variables to predict unknown or future values of other variables.DescriptionFind human-interpretable patterns that describe the data.From Fayyad, et.al. Advances in Knowledge Disc

9、overy and Data Mining, 1996Data Mining Tasks.ClassificationClusteringAssociation Rule DiscoverySequential Pattern DiscoveryRegressionDeviation DetectionClassification ExamplecategoricalcategoricalcontinuousclassTestSetTraining SetModelLearn ClassifierClassification: ApplicationDirect MarketingGoal:

10、Reduce cost of mailing by targeting a set of consumers likely to buy a new cell-phone product.Approach:Use the data for a similar product introduced before. We know which customers decided to buy and which decided otherwise. This buy, dont buy decision forms the class attribute.Collect some related

11、information about the customers.Type of business, where they stay, how much they earn, etc.Use this information as input attributes to learn a classifier model.Clustering DefinitionGiven a set of data points, each having a set of attributes, and a similarity measure among them, find clusters such th

12、atData points in one cluster are more similar to one another.Data points in separate clusters are less similar to one another.ClusteringEuclidean Distance Based Clustering in 3-D space.Intra-cluster distancesare minimizedInter-cluster distancesare maximizedClustering: ApplicationDocument Clustering:

13、Goal: To find groups of documents that are similar to each other based on the important terms appearing in them.Approach: To identify frequently occurring terms in each document. Form a similarity measure based on the frequencies of different terms. Use it to cluster.Gain: Information Retrieval can

14、utilize the clusters to relate a new document or search term to clustered documents.Illustrating Document ClusteringClustering Points: 3204 Articles of Los Angeles Times.Similarity Measure: How many words are common in these documents (after some word filtering).Association Rule DiscoveryGiven a set

15、 of records each of which contain some number of items from a given collection;Produce dependency rules which will predict occurrence of an item based on occurrences of other items.Rules Discovered: Diaper, Milk - BeerAssociation Rule Discovery: Application 1Supermarket shelf management.Goal: To ide

16、ntify items that are bought together by sufficiently many customers.Approach: Process the point-of-sale data collected with barcode scanners to find dependencies among items.A classic rule If a customer buys diaper and milk, then he is very likely to buy beer.So, dont be surprised if you find six-pa

17、cks stacked next to diapers!RegressionPredict a value of a given continuous valued variable based on the values of other variables, assuming a linear or nonlinear model of dependency.Greatly studied in statistics, neural network fields.Examples:Predicting sales amounts of new product based on adveti

18、sing expenditure.Predicting wind velocities as a function of temperature, humidity, air pressure, etc.Time series prediction of stock market indices.Deviation/Anomaly DetectionDetect significant deviations from normal behaviorApplications:Credit Card Fraud DetectionNetwork Intrusion DetectionChallen

19、ges of Data MiningScalabilityDimensionalityComplex and Heterogeneous DataData QualityData Ownership and DistributionPrivacy PreservationStreaming DataMy hope數(shù)據(jù)挖掘研究已經(jīng)開展了近15年。推進該技術的廣泛應用：1. 企業(yè)界已經(jīng)開始關注數(shù)據(jù)挖掘技術研究部門應該做什么？2. 自身技術的研究：易用性可用性3. 與應用領域的結合：金融業(yè)生物信息學信息檢索。飛行器故障診斷與預測、可靠性、My research in recent years1. Mining Acceleration-like Association Rule2. Interior-oriented Intrusion Detection System Based on Multi-agents 3. Fuzzy Clustering Algorithm4. A Fast Trajectory Clustering Algorithm with SamplingMy research in recent years5. An improved C