Capgemini-釋放對人工智能的信心-凱捷Generative AI實驗室的一本劇本（英）

CONFIDENCE

INAI

APlaybookbyCapgeminiGenerativeAILab

2024

SUCCESSFUL,CONFIDENT

ADOPTIONOFAIRELIES

NOTJUSTONCREATING

AITHATWORKS,BUTON

CREATINGAITHATWORKS

RELIABLY,AITHAT’S

ALIGNEDTOHUMAN

EXPECTATIONS,ANDAI

THATWORKSINPEOPLE’S

BESTINTERESTS.

2GenAILab2024

3

TABLEOF

CONTENTS

4GenAILab2024

AITHATWORKS

ProvenAccuracy08

AITHATWORKSRELIABLY

Robustness

10

Dependability

12

Stability

14

AITHAT’SALIGNEDTOHUMANEXPECTATIONS

Sensibility

16

Humility

18

FailsGracefully/ExtrapolatesSensibly

20

Explainability

22

AITHATWORKSINPEOPLE’SBESTINTERESTS

Fairness24

Sustainability26

Privacy28

MARKROBERTS

DeputyHeadGenerativeAILab

Editorinchief

ROBERTENGELS

HeadGenerativeAILab

Editorinchief

AI:Beinggoodwastheeasybit.Nowweneedtobeuseful

Artificialintelligence(AI)issuddenlyeverywhere.Powerfulcontent-generationservicesthatmighthavebeenviewedasbeingfromtherealmofsciencefictionjust12monthsagoarenowabigpartofconversationfromtheboardroomtotheschoolplayground.

Onehugefactorinthisupswingininterestistheriseof

GenerativeAI.Duringthepast12months,theemergenceofhigh-profileGenerativeAIserviceshaspushedAItothefrontpages.WhereAIwasonceperceivedasanicheareaoftechnology,it’snowbeingusedbyallkindsofpeopleforallkindsofuses,whetherit’saskingquestions,writingtext,orgeneratingphotosandcode.

However,don’tconfusetherapidriseofGenerativeAIwitharevolution.WhileaneffectiveuserinterfacelikeChatGPTdemocratizesaccesstopowerfullargelanguagemodels,

themovetowardsAI-poweredserviceswashappening

anyway.Today’sinterestinGenerativeAIissimplythevisiblemanifestationofabehind-the-scenesevolutionthat’sbeenmanyyearsinthemaking.

What’smore,thosedecadesofexperiencegiveusa

proveninsightintothecriticalsuccessfactorsthatmustbeconsideredifwearetoturnaninterestinall-thingsAIintosomethingthathasgenuinecommercialvalue.

Understandingthe

scaleofinvestment

Aswellasthehigh-profilegenerativeservicesthatdominatethenewsagenda,there’sdiversearrayofotherAIproductsandservicesthatarebeingannounced,launchedand

marketedeveryday.ResearcherIDCreportsthatglobal

spendingonAI,includingsoftware,hardware,andservices,will

reach$154billionin2023

,anincreaseof26.9%ontheamountspentduring2022.

ThetechanalystsaysthecontinuedinvestmentinAIwill

meanspendingsurpasses$300billionin2026.Thiscashisalreadyfundingabroadrangeofproof-of-conceptprojects.Whetherthey’reusingAItoimprovecustomerservices,

solvehardscienceandengineeringproblemsoridentifyfraudulenttransactions,companiesareinvestingbillionsofdollarsinrelativelynewtechnologytotryandgain

competitiveadvantageovertheirrivals.

Fromtheoutsidelookingin,thisinvestmentinAIlooks

likeagreatsuccessstory.Thefundingwillcreateproducts

andservicesthathelpshapethefutureoftechnology

andbusiness.Yetthere’sadownside,too–likeallnew

technologywaves,notalloftheseinvestmentswillpayoff.WeseethiseffectacrossCapgemini’sbroadcustomerbase.ManyAIprojects,evenonesthatareapparentlysuccessful,

donotescapetheproof-of-conceptstages.VarioussurveysinrecentyearsputthefailurerateofAIprojectsashighas80%.Whatemergesisacontradiction:whilemanyorganizationsbelieveabiginvestmentinAIwillbecommerciallypositive,largenumbersoftheseprojectsarenotnecessarilypayingoff.So,howcanwereconcilethesetwoverydifferentviewsandcreatecommerciallyusefulAIinitiatives?

5

Changinghowwe

measuresuccess

Thekeychallengeweneedtoovercomeisthatwe’re

allmeasuringthesuccessofAIprojectsinthewrong

way.Whetherit’speoplewhoareusingAI,specialists

developingtools,orthemedia,analystsandinvestors,

we’realllockedintoacollectivedelusionthataccuracyistheonlythingthatmatters.

Successistoooftenmeasuredintermsofhavinghigh

accuracyonnarrowbenchmarktests,orbeingimpressiveorentertaining,whileothercrucialsuccessfactors–areignoredbecausethey’renotwell-understood,excitingorheadline-grabbing.

WhenanAIsystemdoessomethingcorrectly,whether

that’sasimpleclassificationperformedbyatraditional

machine-learningsystem,oraGenerativeAItoolansweringaquestioncorrectly,weattachalotofsignificancetothisaccuracy.Infact,weoftenbaseourentireopinionofthe

systemonthissinglemeasureofaccuracy.

Accuracyissoreveredthateverydayweseebreathlessheadlinesdeclaringthatnewsystemshaveachieved

highlevelsofaccuracyonaparticularproblem.Figuresof“90%accurate”or99%or99.9%arethrownaround–

themore9sthebetter,suchistheobsessionwithhigh

levelsofaccuracy.Toexpertsinthefield,however,this

obsessionwithaccuracyisbothna?veandunhelpful,

asitdrawsattentionawayfromthefactorsthatreally

matterforlong-termsuccess.Inthemajorityofreal-

worlddeployments,howbadlyandAIsystemfailsisfar

moreimportantthathowoftenitsucceeds.Inreality,

anAIsystemthat’s99.99%accuratecouldbedeemeda

completefailureifthe0.001%offailuresarecatastrophic.

Accuracyisnottheonlyimportantfactor–andit’s

certainlynotthemaincauseofmostAIprojectfailures.ThecommercialsuccessofanAIprojectisdependentonacomplexcombinationoffactors,whicharetoooftenignoredorrelegatedtosecondaryconcerns.

However,thesesupposedlysecondaryconcernsare

actuallycriticaltosuccess.Thesefactorsarejustas

importantasaccuracy,maybemoreso,becausetheyareoftentherootcausebehindproblematicbehaviorandfailedAIinvestments.Thesesuccessfactors,whichareoutlinedhere,mustbeconsideredduringthe

developmentandimplementationofanyAIsystemastheywillinstillconfidenceamongthesystem’susersandintheleadersthataredrivingandpayingforit:

AIThatWorks

?ProvenAccuracy–Isgoodatsolvingtheproblem,asmeasuredbybenchmarktests.

AIThatWorksReliably

?Robustness–Handlesunusualormaliciousoutputseffectively.

?Dependability–Alwaysproducesanoutputwithintherequiredtimeframe.

?Stability–Performanceisconsistentanddoesnotdriftovertime.

AIThat’sAlignedtoHumanExpectations

?Sensibility–Makesdecisionsinlinewithhowtheworldorsocietyworks.

?Humility–Understandsitsownlimitations,andrefusestoanswerquestionswhereitdoesn’tknowtheanswer.

?Extrapolatessensibly/Failsgracefully–Actssensiblywhenconfrontedwithscenariosbeyondthoseinwhichitwastrainedandfailssafely.

?Explainability–Canjustifyhowitsolvedtheproblemratherthanworkingasamysteriousblackbox.

AIThatWorksinPeople’sBestInterests

?Fairness–Non-biased.Isequallyfairtoallsub-groups.

?Sustainability–Minimizesharmfulimpactsfromtrainingandongoinguse.

?Privacy–Protectsthesensitivedatathatitwastrainedon.

Conclusion:MakingAIusefulforeveryone

Weseenowthatactuallysolvingataskaccuratelyisjustoneof12equallyimportantfactorsthathelpeveryonetofeel

muchmoreconfidentabouttheAIproductsandservicestheyuse.

Weshouldn’tmakethemistakeofthinkingelements

likehumility,sustainabilityandreliabilityaretheboring

secondaryelementsofanAIendeavor.Whilefocusingonthesefactorswon’tcreatetheexcitementthatcomesfromanAI-generatedimageoressay,itwillensuretheoutputsyourbusinesscreatesaretrustedanduseful.Andonce

thathappens,overtime,thechancesoffailurewillreduce,thelevelsofadoptionwillincrease,andthelikelihoodofcommercialsuccesswillberaisedsignificantly.

AsAIplaysanever-increasinglyimportantroleinourlives,

peoplemustfeelconfidentinthesolutionstheyuse.

Ensuringthese12factorsarealwaysconsideredwillmeanyourbusinessdeliverssignificantcommercialvaluefromAI.Inthisplaybook,wewilldiscusseachofthese12factorsinmoredetail.

6GenAILab2024

Thingspeoplenormally

focusoninAI

Refusingtoanswer,oratleastreportingwhenitdoesn’tknowsomething

ExtrapolatesSensibly

willdosomethingsensible

whenconfrontedwithunseen

databeyondtheboundsof

whatitwastrainedon

Robustness

Willhandle

unusualormalicious

inputswell

Privacy

willnotleak

sensitivedataitwas

trainedon

ProvenAccuracy

Isitgoodatsolvingtheproblem,asmeasuredbytests?

FailsGracefully

Ifitfails,willitfailinasafe&sensibleway?

Sustainability

Impactoftraining

andongoinguseis

notharmful

Explainability

Canitexplain/justify

howitsolvedtheproblem?

-AIthatworks

CONFIDENCE/

TRUSTINAN

AISOLUTION

Thingswenow

recognizearecrucialtomakeAIsuccessful

Stability

performancewill

notunknowinglydrift

overtime

Sensibility

Makesdecisionsinlinewithhowtheworld/nature/physics/cultureworks

Dependability

Willalwaysproduce

anoutput,inthe

requiredtimeframe

Humility

Fairness

Outputisnotbiased

againstanysub-groups

AIthat’salignedwithhumanexpectations

AIthatworksreliably

AIthatworksinpeople’s

bestinterests

7

TIJANANIKOLI?

EXPERTINRESIDENCE

PROVEN

in

ACCURACY

WhendowegettosaythatAIisgoodenough?Whatdoes“good”evenmean?

tobasedecisionsonasthedifferentaccuracymeasuresweusecandramaticallyinfluencehowweinterprettheiroutputs.

Itisimperativetoalsoconsiderreal-worlddimensions.Amodel

mightperformexceptionallyintestsbutfailprofoundlywhen

appliedtoreal-worldscenarios.Thisdiscrepancyhighlightsthe

importanceofacomprehensivedefinitionofgoodness—onethatincorporatesvariousfacetssuchasethicalimplications,social

impact,andalignmentwithhumanvalues.

GenerativeAIhasthrustAIintothespotlightinsectorsfrom

creativeartstodataanalysis,andcustomerservicetoengineering.

However,thisrapidrisehasbroughttoprominencealong-standing

questioninAI:WhatdoesitmeanforAItobe“good”?Traditionally,

theperformanceofmachinelearningmodelshasbeenassessed

onlythroughnarrowmeasuresoftestandvalidationscores.

However,thenewfocusonGenerativeAIwithitscreativityand

hallucinationshasforcedustoreconsiderwhataccuracyreally

meansorwhetheraccuracyisevenrelevantinthisnewworld.

Simplisticmeasuresofaccuracyarenolongergoodenoughforus

8GenAILab2024

WHY?

?AnyonewhoisinvolvedindecisionmakingaroundAIneedstounderstanditsperformance.Thisistruebothofthe

usersofasystem,andofthepeopledesigning,buildingandfundingit.

?Thisneedtounderstandperformancemakesithighlydesirabletocreateasingle,easilydigestiblenumber–accuracy,whichrepresentsthatperformanceprofile.

?However,inalmostallcases,nosinglenumbercantell

youthewholestoryofhowamachinelearningsystem

performs,soweneedoftenneedtousemultiplemetricstodescribetheperformanceprofile.

?Evenifwecouldcapturethehow“good”amodelisinasinglenumber,thatisnotenoughas“good”isasubjectiveterm.

?UnderstandingthemultifacetedessenceofwhatsuccessinAIlookslikeispivotalduetothepotentialconsequencesoffocusingtoomuchonanyonefacet.

?Insomecases,focusingonthewrongtypeofaccuracycancausereal-worldharm.Forexample,astudyof

breast-cancerscreeningintheUKshowedthatana?ve

focusonthewrongsortofaccuracyledtoover-diagnosesandmanywomenunnecessarilyundergoingpainfulandstressfultreatments.

WHAT?

?ConsiderasimplemeasureofaccuracyforanAI

computervisionsystemclassifying100objects,eitherapplesororanges.Wecouldcalculatetheaccuracy

ofthatsystembyjustmeasuringthepercentageofclassificationsthatarecorrect.

?However,thispercentagewouldonlybeausefulmeasureiftherewereexactlythesamenumberofitemsin

bothclasses.If,howeverthereweremoreapplesthan

oranges,asimplepercentageaccuracyfigurewouldnotaccuratelyreflecttheperformanceoftheclassifier.Inanextremecase,iftherewere99applesandoneorange,

andtheclassifieralwayssaid“apple”it’sna?veaccuracy

wouldbe99%,eventhoughithadnoabilitytodetectthedifferencebetweentheclasses.

?Forthisreason,morecomplexstatisticalmeasuresare

used,oftensuchasprecision&recall,orsensitivity&

specificity.Thesemeasuresdescribedifferentfacetsof

accuracy,showinghowwellitperformsinbothitspositiveandnegativepredictions,repeatablyovermultipleuses.

?However,evenusingthesemoresophisticatedmeasuressuchasaccuracy,precisionandrecalldoesnotmeanyourmodel’sreal-worldsuccessisguaranteed.

?Infact,aswewillshowinthisPlaybook,accuracyon

benchmarktestsisonlyoneofmanyequallyimportant

facetsofsuccessthatmustbeconsideredinordertonotjustbesuccessfulonpaper,buttohavegenuinereal-worldsuccesswithuserswhoareconfidentinthatsystem.

RECOMMENDATIONS

?First,ensureyouaremeasuringandcommunicating

accuracyeffectively.Itisextremelyunlikelythataccuracycanberepresentedbyasinglenumber,sousemore

appropriatemeasurestosetusers’expectationsabouttheperformanceprofileofasystem.

?Don’tusesimplisticmeasuresofaccuracyasthesolecriteriafordeclaringsuccessinanAIsystem.

?EducateeveryoneinthebusinessabouthowtotalkaboutaccuracyinAIsystems.Striveforaculturewhereeveryone,rightuptotheboardroom,iscomfortableaskingquestionsaboutsensitivityandspecificity,precisionandrecalletc.

?Beyondaccuracy,aholisticapproachisnecessary.

Organizationsmustembracetransparency,ethics,andfairnessintheirAIendeavors.Considerusingaplaybook,likethisone,toremindeveryoneinvolvedinAIsystemsdesigntothinkaboutthemultiplefacetsthatleadto

successfulAI,notjustonaccuracyalone.

?Oneoftheprimarypitfallsisamyopicfocusontechnicalmetrics.Ignoringbiasesintrainingdata,overlooking

ethicalimplications,orneglectingcommunityfeedbackcanleadtocatastrophicoutcomes.Contextualfit,forinstance,cannotbemeasuredeasily.Butisthefinal

definingfactorfor“goodness”

LINKS

?ValidatingLargeLanguageModelswithReLM.Kuschnicketal.CarnegieMellonUniversity,2023.

/

pdf/2211.15458.pdf

?Langchainblogpost:“HowCorrectareLLMEvaluators”,problematizingthepossibilitiestofacilitatemeasurementof“provenaccuracy”.

https://blog.langchain.dev/how

-

correct-are-llm-evaluators/

?GEDLTprojectonprompting,writingstylesandqualityof

answeringTheGDELTProjectisarealtimenetworkdiagramanddatabaseofglobalhumansocietyforopenresearch:

/large-language-models-llms

-

planetary-scale-realtime-data-current-limitations/

9

PROVENABILITY

MITALIAGRAWAL

ROBUST

in

EXPERTINRESIDENCE

WillanAIsystemalwaysrespondtosimilarinputsinaconsistentmanner?Canitcope

withdeliberatemaliciousattacksintheinput?Allofthesequestionsrelatetotheideaofrobustness-ameasureofhowwellanAIsystembehaveswhenthesignalsitreceivesarenotthesameaswhatitwastrainedon.

RobustnessisacornerstoneofreliableAIsystems,ensuringresilienceinthefaceofadversity.Inthedynamiclandscapeofartificialintelligence,twoparamountchallengesarise:dealingwiththehugevariationofinputsasystemwillencounterintherealworldinaconsistentmanneranddefendingagainstdeliberatelymaliciousinputs,oftenmanifestedasadversarialattacks.

UnderstandingandfortifyingAIagainstthesechallengesisessentialinshapingafuturewhereAItechnologiescanbetrustedandreliedupon.

WHY?

.InanerawhereAIisincreasinglyprevalentinourdailylives,robustnessisafundamentalpillaroftrustworthiness.

.Duetotheircomplexitythough,AIsystemsaresusceptibleto

variousvulnerabilities,bothinthealgorithmsandthedatatheyaretrainedon.

.AsimplewaytodemonstrateifanAIsystemisrobustornotistoaskittoperformasimilartasktwice.Providingsignificantlydifferent

answerstothesamequestionwillcausehumanstorapidlylosetrustinthesystem,butmanyAIsystemswillfailthissimpletest.

.Therewillalwaysbeconfusinginputsinthereal-world,and

unfortunatelytherewillalwaysbemaliciousactorswhotryto

deliberatelyaffecttheoutputsofourAIsystems.Eveninthebestcases,withnomaliciousactor,wewillstillforeverbelockedinanarmsracebetweenourmachinelearningmodelsandtheinfinitecomplexitythattherealworldwillthrowatthem.

.Therefore,therewillalwaysbeaneedtouseapproachesto

maximizetherobustnessofourAImodels,andinsomecaseswerequireverifiableproofofthatrobustness.

.Byaddressingthecomplexitiesofunusualdataandadversarial

10GenAILab2024

ROBUST

attacks,wepavethewayforAIsystemsthatnotonlyexcelunderidealcircumstancesbutareresilientinthefaceof

unexpectedinputsanddeliberateattacks.

WHAT?

.Whilstmachinelearningexpertshavelongknownabouttheproblemsofrobustness,GenerativeAItoolsnow

alloweveryonetoseetheextentofthisproblem–evensmallchangesinthephrasingofapromptcanproducecompletelydifferentoutputsandmeanings.

.Deliberatelymaliciousinputs,knownasadversarial

attacks,exploitthevulnerabilitiesofAIsystems,leadingthemtomakeerroneousjudgments.Theseattackscanhavedireconsequences,especiallyinsafety-related

applicationssuchasautonomousvehiclesorhealthcaresystems,somakingrobustdefensesisimperative.

.Adversarialattacksfallintotwomainclasses

.White-boxattackswhichuseknowledgeofthemodeltoachievetheirimpact.

.Black-boxattackswhichdonothaveknowledgeoftheunderlyingmodel.

.Theseattacksmightalsobeuntargeted,wheretheaimistojustachieveanycorruptionoftheoutput,ortargeted,wheretheaimistocoercethemodeltoproduceaspecificoutput.

.Thankfully,maliciousattacksonAIsystemsarerelativelyrare,andthemorecommonproblemiswhereAI

systemsencounteratypical,unfamiliardatainreal-worldscenarios.Thiscanrangefromnovelenvironmental

conditionsforautonomousvehiclestounprecedenteduserinputsinchatbots,challengingthesystem’sabilitytomakeaccuratepredictionsordecisions.

.Whenfacedwithunusualdata,AIsystemsmightexhibitunpredictablebehavior,potentiallyjeopardizingthe

trustabilityoftheoutputs.Ensuringrobustnessinsuchsituationsnecessitatestrainingmodelsnotjustonlargerdatasetsbutonmorediversedatasetsthatencompass

(A)

awidearrayofpossibleinputs,preparingthemforunforeseenscenarios.

RECOMMENDATIONS

.Makesureyouunderstandthescaleoftheproblemin

yourusecase–testyoursystemstomakesurethatsmallchangesintheinputdonotproducesignificantchangesinthemeaningoftheoutput.

.ForLLMsspecifically,guaranteedrobustnessismuch

hardertoachievebecausethesemodelsdonotactuallyunderstandthemeaningofthelanguagetokenstheymanipulate.Whereahumanmightseetwophrases

asbeingthesame,theycouldbeinterpretedinverydifferentwaysbyanLLMandproducingsubstantiallydifferentoutputs.

.Additionally,inputpreprocessingtechniquescould

(B)

enhanceasystem’sabilitytoprovidemorerobustresultsbyensuringmultiplerephrasedversionsofthepromptproduceconsistentoutput.

.Traditionallyusedincybersecurity,redteaminginvolvessimulatingadversarialattackstoidentifyvulnerabilitiesandweaknessesinasystem.WhenappliedtoAI,red

teamingservesasapotenttooltoassesstheresilienceofmachinelearningmodels,algorithms,andapplicationsagainstmaliciousintentandunexpectedinputs.

.Wecanalsouseothermachinelearningsystemsasaredteam,exploitingmalicioustechniquesforpositiveuseinanapproachcalledadversarialtraining.Inthisapproachmodelsareexposedtoadversarialexamplesduring

training,enablingthemtorecognizeandresistsuch

inputs.Thisapproachpitsonemachinelearningsystemagainsttheother,resultinginbothbeingbetterandtheoverallsystembeingsignificantlymorerobust.

.Insomecases,itmaybepossibletouseverifiably

robustapproachestotraining,suchasIntervalBound

Propagation(IBP),whichcanguaranteecertainlevelsofrobustness,althoughoftenattheexpenseofaccuracyi.e.overallaccuracymaybelower,butyoucanbesurethat

whenitdoesmakeapredictionitiscorrect.

Imagesshowingconfusingdataan

AIvisionsystemmightencounterin

thereal-world,sometimesnaturally

occurring,sometimesasaresultof

maliciousattacks.

Anexampleofhowaseeminglyinconsequentialreframing

oftheinputtoanLLMcanproducesubstantiallydifferent,

andincorrect,output.

11

WEIWEIFENG

DEPENDABILITY

in

EXPERTINRESIDENCE

WhilstmostofthepropertiesdescribedinthisplaybookrelatetothecontentandqualityofanAIsystem’soutput,weoftenneglectsomeoftheoperationalconsiderationsfor

deployingAIinreal-worldsituations.Oneofthemostimportantoftheseisdependability–willanAIsystemactuallygiveusananswerwhenweneedit?

AswestarttomoveAIsystemsfromthelabtothereal-world,one

ofthepracticalrealitiesthatwemustconsideristiming.Inmany

cases,thespeedofanAIsystem’sresponseiscrucial.Itdoesn’treallymatterifacustomer-servicechatbottakes10secondstorespond,butitwouldclearlybeabigproblemifanautonomousvehicletook10

secondstoconsideritsactionswhilstdrivingatspeedonaroad.

Thispresentsanimportantanddifficultdilemmatosolve.ModernAIhasachievedmanyimpressiveresults,butthisislargelypoweredby

hugeneuralnetworkmodelswhichareslowtoexecuteandrequirelevelsofcomputepowerthatarenormallynotavailableinreal-worlddeploymentsofAIsystems.Ifwedon’thaveguaranteesthatanAI

systemwillrespondasquicklyasweneeditto,thenconfidence

andadoptionwillfalter.Fundamentallydifferentarchitecturesarerequiredwheretimelinessofresponseisimportant.

12GenAILab2024

DEPENDABILITY

WHY?

?Inreal-worlddeploymentsofAI,timingmatters.Ahigh-performingmodelwithgoodaccuracyisworthlessifit

doesn’trespondquicklyenoughforitsoutputtobeused.

?Thisismostobviousinsafety-criticalandreal-timecontrolsituationswherenon-negotiableguaranteesonresponsetimesarepresent.

?Eveninsituationsthatarenotsafety-related,responsetimecandramaticallyaffecttheuserexperiencetothepointthattheymightloseconfidenceinasystemthatdoesn’trespondquicklyenough.

?Inothersituations,lowlatencyresponsesarerequiredforreasonsofresponsivenessandthroughput.Longdeliberationinpursuitoftheperfectanswerinthesetypesofproblemscouldcausewidespreaddisruption.Forexample–

?Incredit-cardfrauddetection,wherevastnumbersof

transactionsmustbeassessedquicklytopreventdelays.

?InAI-supportedemergencyresponsesystems,wherestressandtheneedforswiftinformationavailabilityhaveadirectimpactonoutcomes.

?Inreal-timeschedulingproblems,suchastraffic-lightscheduling,elevatordispatchingetc.

?Dynamicadvertisementselectiononwebsites,whereaslowdecisionwouldruintheuser-experienceofthehostwebsite.

WHAT?

?ThespeedofanAIsystem’sresponsehasalwaysbeena

primaryconsiderationinAIresearch,asmanyoftheclassicbenchmarksofartificialintelligencehaveatimingelementtothem–playinggames,havingconversations,driving

vehicles,interactiverobotsetc.

?Inmanycasesitmaybepossibletosolveamachine

learningtaskwithgoodorevenperfectaccuracyiftimingisnotanissue,butsolvingtheengineeringproblemof

deployingthatsamemodelintoamoreconstrainedandtime-criticalenvironmentmaybeimpossible.

?Insomecases,itmaybepossibletocompressorprunealargemodel,toimproveitsresponsetime.ThisisalreadycommonplaceinmanyEdgeAIdeploymentsinorder

tosqueezemoreperformanceoutoflimitedhardware.However,whilstthisapproachimprovesperformance,itcannotguaranteeperformance.

?Mostmachinelearningmodelsareeffectivelynon-

deterministic,meaningthattheexecutionofthe

model(inference)willneverbepredictable.Therefore,ifguaranteesofperformancearerequired,then

justshrinkingabigmodelwillneverbetheanswer.Fundamentallydifferentarchitecturesarerequired.

?Themostobviousandwell-knownarchitectureisthe

so-calledclassifier-cascade.Inthiscase,machinelearning

modelsarearrangedinacascade,startingwithextremelysimpleandsmallclassifiersthatcanprovideaquickanswerimmediately.Iftimeallows,processingpassesontoamorecomplexbuttime-consumingclassifier,andthisprocess

continuestothebottomofthestack.Thisarchitecturemeansthatananswercanbewecaninterruptthe

processingatanypointandgetananswer.

?Thisissimilartowhatweseeinhumandecisionmaking,wherewehavefast“System1”thinkingtogivean

immediateresponse,followedbyslowerandmore

deliberate“System2”thinking.Inthecaseofclassifiercascades,theremaybehundredsoflevels,iterativelyrefiningtheanswerasfarastimeallows.

?Inmostcases,thefirstlevelofsuchacascadewouldbeanon-AIsystem,whichencodesbasicdefaultbehavior.

?Theperformanceofsystemscanbeenhancedthroughtieredapproaches.Ateachtier,thesolutionshould

beevaluatedagainsttheprevioustiertodetermineifitprovidesasignificantimprovement.Thisevaluationprocessallowsforearlytermina