Topic7PartIMeasurementandEvaluation第七講第一部

1、Topic 7 : Part IMeasurement and Evaluation第第七七講：講： 第一部份第一部份量度及評估量度及評估Paul GandelSyracuse UniversitySingapore Management UniversityMethods to Measure and Display Outcomes量度方法及展示結(jié)果量度方法及展示結(jié)果2Basics of Measurement and Evaluation量度及評估的基本因素量度及評估的基本因素 Defining measurement and evaluation Frameworks for meas

2、urement Outcomes-based evaluation Balance scorecard approach3 為量度及評估下定義為量度及評估下定義 量度架構(gòu)量度架構(gòu) 基于結(jié)果的評估基于結(jié)果的評估 平衡計分卡法平衡計分卡法Why Measure?為何要量度？為何要量度？ Reference pointqWhere service was in the pastqCurrent situationqFuture trends Informs qDecision makersqManagersqStaff Sets Priorities4 參考點參考點q過往的服務(wù)過往的服務(wù)q目前的景況

3、目前的景況q未來的趨勢未來的趨勢 報告報告 q決策者決策者q經(jīng)理經(jīng)理q員工員工 設(shè)定優(yōu)次序設(shè)定優(yōu)次序Problem 問題問題 Not too little data NOT THE RIGHT DATA!5 數(shù)據(jù)數(shù)據(jù)量不宜太少量不宜太少 不是正確的數(shù)據(jù)！不是正確的數(shù)據(jù)！Measurement vs. Evaluation量度量度與與評估評估 Measurement is collecting the raw numbers Evaluation is the determination of the value of the service Evaluation requires a pers

4、pective Creating indicators allows for incorporating some context6 量度是收集原始數(shù)目量度是收集原始數(shù)目 評估是決定服務(wù)的價評估是決定服務(wù)的價值值 評估需要一套正確的評估需要一套正確的方法方法 設(shè)定指標時容許加入設(shè)定指標時容許加入某些內(nèi)容某些內(nèi)容Measurement Framework量度架構(gòu)量度架構(gòu)7From Joseph Mathews Measuring for ResultsMeasurement Framework量度架構(gòu)量度架構(gòu)Resources資源Utilization效用Capability能力Impact &

5、Effect影響及效果InputMeasures投入量度OutputMeasures產(chǎn)出量度ProcessMeasures過程量度Outcomes結(jié)果DemandValue價值Quality質(zhì)量8Input Measures投入量度投入量度 Measure the capacity/potential Easy to collect and report Examples:qIncome/ExpendituresqStaffingqCollection (total, growth, recency)qInfrastructure (cost, speed, space, number of

6、workstations, etc.)qUser base statistics9 量度能力或潛力量度能力或潛力 容易收集及報告容易收集及報告 范例范例:q收入收入/ 支出支出q人力資源人力資源q館藏館藏 (總數(shù)，增長，最總數(shù)，增長，最近購買的近購買的)q基礎(chǔ)設(shè)施基礎(chǔ)設(shè)施(成本，速度，成本，速度，空間，空間，工作間數(shù)目等工作間數(shù)目等)q基于用戶的統(tǒng)計基于用戶的統(tǒng)計Process/Efficiency Measures過程過程 / 效率量度效率量度 Are we doing things correctly? Efficiencyhow economical is an activity? Co

7、st per transaction for reference service, document delivery, etc. Productivityfocus on time to complete task or activity. Time to catalog an item, time to process a transaction, etc. System Reliabilityfocus on percent of uptime or availability of the system10 我們是否做得正確我們是否做得正確? 效率效率 活動是否夠經(jīng)濟？活動是否夠經(jīng)濟？每

8、項咨詢參考及文獻送每項咨詢參考及文獻送遞等服務(wù)的成本遞等服務(wù)的成本 生產(chǎn)力生產(chǎn)力 著眼于完成任著眼于完成任務(wù)或活動的時間。將一務(wù)或活動的時間。將一項館項館藏藏分類的時間，處分類的時間，處理一項服務(wù)的時間。理一項服務(wù)的時間。 系統(tǒng)可靠系統(tǒng)可靠性性 著眼于正著眼于正常運作時間的百分比或常運作時間的百分比或系統(tǒng)可用性系統(tǒng)可用性O(shè)utput Measures產(chǎn)出量度產(chǎn)出量度 Looks at the use of the Library or a particular service Services. Counts and use per capita statistics. Annual circ

9、ulation, attendance, etc Qualitywhat was provided and how was it provided. Lead to measures of satisfaction and value.11 考慮圖書館或某項專門考慮圖書館或某項專門服務(wù)的用量服務(wù)的用量 服務(wù)。以人均統(tǒng)計方法服務(wù)。以人均統(tǒng)計方法計算。書籍流通量，出計算。書籍流通量，出席率等。席率等。 質(zhì)量質(zhì)量 有何服務(wù)及如何有何服務(wù)及如何提供提供該該服務(wù)。帶出滿意服務(wù)。帶出滿意度及價值的量度。度及價值的量度。Satisfaction Measures滿意度量度滿意度量度 Blanket ques

10、tion = useless information Ask about specific components of the experience Ask about how the user is doing not just how the service is doing Ask about expectations (LIBQUAL) Ask about importancewas this service important to the user?12 一般的問題一般的問題= = 沒用的信息沒用的信息 詢問某經(jīng)驗的特別部分詢問某經(jīng)驗的特別部分 詢問用戶的感受如何，而詢問用戶的感受

11、如何，而不單只是詢問服務(wù)如何不單只是詢問服務(wù)如何 了解期望了解期望(LIBQUAL)(LIBQUAL) 詢問重要性詢問重要性 該項服務(wù)該項服務(wù)對用戶是否重要對用戶是否重要? ?Priority and Performance優(yōu)先次序優(yōu)先次序及表現(xiàn)及表現(xiàn) What priority should the library give to the following? How important? How well does the library perform the following services or functions? Usually use a 7 point Likert sc

12、ale13 圖書館如何為以下項目定圖書館如何為以下項目定優(yōu)先次序？優(yōu)先次序？ 有多重要？有多重要？ 圖書館在以下服務(wù)或圖書館在以下服務(wù)或功能功能表表現(xiàn)如何現(xiàn)如何？ 通常用通常用 Likert Likert 的的 7 7 級分表級分表14Success Metrics: Purpose成功規(guī)范：目的成功規(guī)范：目的 To establish clear accountability and clarity of purpose Both measured group and the group for whom measures are collected are clearly stated T

13、o create focus and alignment The measures collected are in line with the mission, goals, vision, and values of the libraryPartially from Matthews, J. (2004) Measuring for Results: The Dimensions of Public Library Effectiveness設(shè)立清晰的責任和明確的設(shè)立清晰的責任和明確的目的目的明確定義被量度的小組及征明確定義被量度的小組及征用資料的小組用資料的小組重點設(shè)定及調(diào)整重點設(shè)定及

14、調(diào)整收集的數(shù)據(jù)應與圖書館的使收集的數(shù)據(jù)應與圖書館的使命、目標、愿景及價值配合命、目標、愿景及價值配合15Success Metrics: Purpose成功規(guī)范：目的成功規(guī)范：目的 To enable continuous refinement Measures are reviewed and changed to keep up with changes in environment To provide rigorous indicators Measures are well-defined using unambiguous language and with as little b

15、ias as possiblePartially from Matthews, J. (2004) Measuring for Results: The Dimensions of Public Library Effectiveness 容許持續(xù)改善容許持續(xù)改善 檢討及改變數(shù)據(jù)以配合環(huán)檢討及改變數(shù)據(jù)以配合環(huán)境轉(zhuǎn)變境轉(zhuǎn)變 提供嚴密的指標提供嚴密的指標 量度指量度指標標應用不含糊的語應用不含糊的語言作嚴格的定義以盡量減言作嚴格的定義以盡量減少偏見少偏見16Balancing Success Metrics平衡成功規(guī)范平衡成功規(guī)范Absolute versus relative measures

16、Comparisons add context (but are not always appropriate)Objective versus subjective How much does background of evaluator affect the interpretation of the results?Outcomes over process Focus on outcomes, not on the process Performance versus diagnostic Measure achievements and discover problematic a

17、reasPeer institution benchmarks versus best practicePartially from Matthews, J. (2004) Measuring for Results絶對指標與相對指標絶對指標與相對指標 Comparisons add context (但并不總是適合但并不總是適合)客觀與主觀客觀與主觀 評估人的背景對結(jié)果的演繹影評估人的背景對結(jié)果的演繹影響有多深響有多深結(jié)果與過程結(jié)果與過程 側(cè)重于結(jié)果而非過程側(cè)重于結(jié)果而非過程 表現(xiàn)及診斷法表現(xiàn)及診斷法 量度成就及發(fā)現(xiàn)有問題的地方量度成就及發(fā)現(xiàn)有問題的地方同水平學院的標準與優(yōu)良慣同水平學院的標

18、準與優(yōu)良慣例例17Types of Performance Indicators表現(xiàn)指標表現(xiàn)指標Direct versus indirect indicators Measuring a specific activity vs. measuring a surrogate for an activity (looking at citations as surrogate for measuring value of materials)Lagging versus leading indicators Lagging = actual performance, while leading

19、= prediction of something elseSocial versus economic indicators Measures indicative of the impact of the library on a community as compared to the economic importance (and cost) of the libraryFrom Matthews, J. (2004) Measuring for Results: The Dimensions of Public Library Effectiveness直接與間接指標直接與間接指標

20、 量度一項特定的活動與量度一量度一項特定的活動與量度一項相關(guān)的活動項相關(guān)的活動 (考慮以引用次數(shù)考慮以引用次數(shù)作為量度實際價值的相關(guān)指作為量度實際價值的相關(guān)指標標)落后的與領(lǐng)先的指標落后的與領(lǐng)先的指標 落后落后 =實際表現(xiàn)實際表現(xiàn),而領(lǐng)先而領(lǐng)先 =其他其他預測預測社會指標與經(jīng)濟指標社會指標與經(jīng)濟指標 此量度方法表示出與圖書館的此量度方法表示出與圖書館的經(jīng)濟重要性經(jīng)濟重要性(及成本及成本)相比圖書館相比圖書館對社會的影響對社會的影響18Balanced Scorecard Approach平衡計分卡平衡計分卡User perspective How do users perceive the l

21、ibrary? Is that perspective changing?Internal perspective In which services should the library aim to excel?Information resources perspective What should be in the library collections? Innovation perspective How can the library continue to develop new services?Financial perspective How can the libra

22、ry demonstrate the best value for the money invested by the institution in the library?Based upon: Matthews, J. (2004) Measuring for Results: The Dimensions of Public Library Effectiveness 用戶觀感用戶觀感 用戶對圖書館有何用戶對圖書館有何觀感觀感? ? 他們他們的觀感是否在轉(zhuǎn)變的觀感是否在轉(zhuǎn)變? ? 內(nèi)部認知內(nèi)部認知 圖書館應著意突出哪幾項服務(wù)？圖書館應著意突出哪幾項服務(wù)？ 資源概念資源概念 圖書館藏應包含

23、甚么？圖書館藏應包含甚么？ 創(chuàng)新觀念創(chuàng)新觀念 圖書館如何繼續(xù)新服務(wù)？圖書館如何繼續(xù)新服務(wù)？ 財務(wù)視角財務(wù)視角 圖書館如何展示學院對圖書館圖書館如何展示學院對圖書館的投資物有所值？的投資物有所值？19Mission 使命Process Perspective流程構(gòu)面Financial Perspective財務(wù)構(gòu)面Customer Perspective顧客構(gòu)面Balanced Scorecard 平衡計分卡平衡計分卡 Library Perspective 以圖書館為例以圖書館為例InformationResourcesPerspective資源構(gòu)面Organizational Dashboa

24、rds組織儀表盤組織儀表盤 User-friendly tool to display performance Effective and meaningful way to track progress View results: efficiency, effectiveness, and impact Should integrate into the strategic plan a limited number of critical indicators of overall library performanceqDepartments may have their own da

25、shboards20Source: Butler, L. The Nonprofit Dashboard: A Tool for Tracking Progress (BoardSource, 2007) 容易使用的表現(xiàn)展示工具容易使用的表現(xiàn)展示工具 有效及有意義的進度追蹤有效及有意義的進度追蹤方法方法 看得見的結(jié)果：效率、效看得見的結(jié)果：效率、效力及影響力及影響 將圖書館整體表現(xiàn)的某些將圖書館整體表現(xiàn)的某些關(guān)鍵指標納入策略規(guī)劃關(guān)鍵指標納入策略規(guī)劃q各部門可有自己的儀表各部門可有自己的儀表盤盤What to measure?qOutcomesqMission as spineqStrategi

26、c initiativesqDrivers of successqRisk factorsqService/resource matrixHow to track and display?qCombine statistics with visual indicatorsqGraphs and21Source: Butler, L. The Nonprofit Dashboard: A Tool for Tracking Progress (BoardSource, 2007)Organizational Dashboards / 組織儀表盤組織儀表盤量度甚么？量度甚么？結(jié)果結(jié)果使命是支持因素

27、使命是支持因素策略上的主動權(quán)策略上的主動權(quán)成功動力成功動力冒險因素冒險因素服務(wù)服務(wù)/資源矩陣資源矩陣如何追蹤及展示如何追蹤及展示將統(tǒng)計數(shù)字與看將統(tǒng)計數(shù)字與看得見得見的的指標結(jié)合指標結(jié)合圖形及圖表圖形及圖表Relating the Mission to the Dashboard將使命與儀表盤聯(lián)系起來將使命與儀表盤聯(lián)系起來Mission Statement /使命宣言使命宣言Purpose目的目的Audience讀者讀者Methods方式方式Outcomes結(jié)果結(jié)果The library provides information 圖書館提供信息圖書館提供信息for faculty and stu

28、dents of the university 為大學學系及學生為大學學系及學生through electronic and print resources 透過電子及印刷資源透過電子及印刷資源to enable our clients to meet their research and learning needs以令顧客滿足教研需要以令顧客滿足教研需要Performance Indicators / 表現(xiàn)指標表現(xiàn)指標Number of searches performed using e-resources電子資源的搜尋次數(shù)電子資源的搜尋次數(shù)Use of facilities for research, group and individual study, determined through random counts以隨機數(shù)確定為研究，以隨機數(shù)確定為研究，小組與個人學習而建小組與個人學習而建的設(shè)備的運用的設(shè)備的運用Involvement of faculty and students presenting their research results based upon library res