ROC曲線.ppt

1、ROC曲線,Hongjie Wu,01,Basic concept,02,Plotting an ROC curve,03,AUC,contents,Basic concept,Receiver operating characteristic (ROC) curves are a useful visual tool for comparing two classification models.(比較兩個(gè)分類模型有用的可視化工具) The curve is created by plotting the true positive rate (TPR) against the false

2、positive rate (FPR) at various threshold settings. （ROC曲線顯示了真正例和假正例之間的權(quán)衡）,Basic concept,true positive (TP真正例)診斷為有，實(shí)際上也有高血壓 eqv. with hit true negative (TN真負(fù)例)診斷為沒(méi)有，實(shí)際卻沒(méi)有高血壓 eqv. with correct rejection false positive (FP假正例)診斷為有，實(shí)際上也沒(méi)有高血壓 eqv. with false alarm, Type I error false negative (FN假負(fù)例)診斷為沒(méi)

3、有，實(shí)際卻有高血壓 eqv. with miss, Type II error,Basic concept,sensitivity or true positive rate (TPR)Y軸 eqv. with hit rate, recall 在所有實(shí)際為陽(yáng)性的樣本中，被正確地判斷為陽(yáng)性之比率 fall-out or false positive rate (FPR)X軸 在所有實(shí)際為陰性的樣本中，被錯(cuò)誤地判斷為陽(yáng)性之比率,Plotting an ROC curve,Figure 8.18 shows the probability value (column 3) returned by

4、a probabilistic classifier for each of the 10 tuples in a test set, sorted by decreasing probability order. Column 1 is merely a tuple identification number, which aids in our explanation. Column 2 is the actual class label of the tuple. There are five positive tuples and five negative tuples, thus

5、and . As we examine the known class label of each tuple,we can determine the values of the remaining columns, TP, FP, TN, FN, TPR, and FPR. We start with tuple1, which has the highest probability score, and take that score as our threshold, that is, . Thus, the classifier considers tuple 1 to be pos

6、itive, and all the other tuples are considered negative. Since the actual class label of tuple 1 is positive, we have a true positive, hence and . Among the remaining nine tuples, which are all classified as negative, five actually are negative (thus, ). The remaining four are all actually positive,

7、 thus, . We can therefore compute , while .Thus, we have the point for the ROC curve.,Plotting an ROC curve,Figure 8.18 Tuples sorted by decreasing score, where the score is the value returned by a probabilistic classifier. (元組按遞減得分排序，其中得分是概率分類器的返回值),對(duì)于二類問(wèn)題， 選擇閾值t， 使得f(X)=t的 元組X視為正， 而其它元組視為負(fù),Plottin

8、g an ROC curve,There are many methods to obtain a curve out of these points, the most common of which is to use a convex hull. The plot also shows a diagonal line where for every true positive of such a model, we are just as likely to encounter a false positive. For comparison, this line represents

9、random guessing（許多方法可以從這些點(diǎn)得到凸包。該圖現(xiàn)實(shí)的對(duì)角線，對(duì)模型的每個(gè)真正例元組，都恰好遇到一個(gè)假正例，為了比較，這條直線代表隨機(jī)猜測(cè)）.,AUC,trapezoid method 簡(jiǎn)單地將每個(gè)相鄰的點(diǎn)以直線連接，計(jì)算連線下方的總面積。因?yàn)槊恳痪€段下方都是一個(gè)梯形，所以叫梯形法。 優(yōu)點(diǎn)：簡(jiǎn)單，所以常用。 缺點(diǎn)：傾向于低估AUC。 ROC AUCH AUC of ROC是機(jī)器學(xué)習(xí)的社群最常使用來(lái)比較不同模型優(yōu)劣的方法 。然而近來(lái)這個(gè)做法開(kāi)始受到質(zhì)疑，因?yàn)橛行C(jī)器學(xué)習(xí)的研究指出，AUC的噪聲太多，并且很常求不出可信又有效的AUC值，使得AUC在模型比較時(shí)產(chǎn)生的問(wèn)題比解釋的問(wèn)題更多 。,AUC,To assess the accuracy of a model, we can measure the area under the curve.（為了評(píng)估模型的準(zhǔn)確率，可以測(cè)量曲線下方的面積） Several software packages are able to perform such calculation.（一些軟件包可以用來(lái)進(jìn)行這些計(jì)算）,AUC,從AUC判斷分類器（預(yù)測(cè)模型）優(yōu)劣的標(biāo)準(zhǔn)： AUC = 1，是完美分類器，采用這個(gè)預(yù)測(cè)模型時(shí)，存在至少一個(gè)