港科大-城市基礎(chǔ)大模型UFM綜述與方案 Towards Urban General Intelligence A Review and Outlook of Urban Foundation Models

1

arXiv:2402.01749v1[cs.CY]30Jan2024

TowardsUrbanGeneralIntelligence:AReviewandOutlookofUrbanFoundationModels

WeijiaZhang,JindongHan,ZhaoXu,HangNi,HaoLiu,SeniorMember,IEEEandHuiXiong,Fellow,IEEE

Abstract—Machinelearningtechniquesarenowintegraltotheadvancementofintelligenturbanservices,playingacrucialroleinelevatingtheefficiency,sustainability,andlivabilityofurbanenvironments.TherecentemergenceoffoundationmodelssuchasChatGPTmarksarevolutionaryshiftinthefieldsofmachinelearningandartificialintelligence.Theirunparalleledcapabilitiesincontextualunderstanding,problemsolving,andadaptabilityacrossawiderangeoftaskssuggestthatintegratingthesemodelsintourbandomainscouldhaveatransformativeimpactonthedevelopmentofsmartcities.DespitegrowinginterestinUrbanFoundationModels(UFMs),thisburgeoningfieldfaceschallengessuchasalackofcleardefinitions,systematicreviews,anduniversalizablesolutions.Tothisend,thispaperfirstintroducestheconceptofUFManddiscussestheuniquechallengesinvolvedinbuildingthem.Wethenproposeadata-centrictaxonomythatcategorizescurrentUFM-relatedworks,basedonurbandatamodalitiesandtypes.Furthermore,tofosteradvancementinthisfield,wepresentapromisingframeworkaimedattheprospec-tiverealizationofUFMs,designedtoovercometheidentifiedchallenges.Additionally,weexploretheapplicationlandscapeofUFMs,detailingtheirpotentialimpactinvariousurbancon-texts.Relevantpapersandopen-sourceresourceshavebeencol-latedandarecontinuouslyupdatedat:

/usail-

hkust/Awesome-Urban-Foundation-Models.

I.INTRODUCTION

UrbanGeneralIntelligence(UGI)referstoacon-ceptualizedadvancedformofartificialintelligencetailoredtounderstand,interpret,andadeptlymanagecomplexurbansystemsandenvironments.AnalogoustoArtificialGeneralIntelligence(AGI),UGIisenvi-sionedtoautonomouslyperformanyintellectualtaskrelatedtourbancontexts,rivalingorevensurpassinghumancapabilities,therebytransformingcitiesintomorelivable,resilient,andadaptivespaces.

Machinelearningtechnologieshavebecomepivotalintrans-formingurbanlandscapes,underpinningthedevelopmentofvarioussmartcityservices.Thesetechnologiesenhanceurbanintelligencebyenablingmoreefficientresourceallocation,

WeijiaZhang,ZhaoXu,andHangNiarewiththeArtificialIntelligenceThrust,TheHongKongUniversityofScienceandTechnology(Guangzhou),Guangzhou,PRC.E-mail:{wzhang411,zxu674,hni017}@connect.hkust-

JindongHaniswiththeAcademyofInterdiscriplinaryStudies,TheHongKongUniversityofScienceandTechnology,HongKongSAR,PRC.E-mail:jhanao@connect.ust.hk

HaoLiuandHuiXiongarewiththeArtificialIntelligenceThrust,TheHongKongUniversityofScienceandTechnology(Guangzhou),Guangzhou,PRCandtheDepartmentofComputerScienceandEngineering,TheHongKongUniversityofScienceandTechnology,HongKongSAR,PRC.E-mail:{liuh,xionghui}@ust.hk

improvedpublicservices,andelevatedqualityoflifeforcity

dwellers[1].Throughanalyzingvastdatasetsfromdiverse

sourcessuchassatellites,urbansensors,andsocialmediaplatforms,machinelearningalgorithmsidentifyintricateurbanpatternsandfacilitateaccurateforecastingofcitydynamics.Suchanalyticalpowerprovesessentialinboostingurbanoperationalefficiencies,suchassmartenergymanagement,trafficflowoptimization,andenvironmentalmonitoring,all

contributingtomoresustainableandlivablecities[2],[3]

.Furthermore,theintegrationofmachinelearningalgorithmsalsoenhancesreal-timeresponsivenessincriticalscenarioslikeemergencyservicesandpublicsafety,therebyredefiningurbanenvironmentsasmoreadaptive,responsive,andintelli-gentsystems.

Therecentadventoffoundationmodels,includingLarge

LanguageModels(LLMs)suchasChatGPT1

andVisionFoundationModels(VFM),hassignificantlyreshapedtheresearchlandscapeinmachinelearningandartificialintelli-

gence[4].Thesemodelsarecharacterizedbytheirextensive

pre-trainingonlarge-scaledatasets,whichimbuesthemwithunparalleledemergentabilities,includingcontextualreason-ing,complexproblemsolving,andzero-shotadaptability

acrossdiversetasks[5],[6].Suchabilitiesmakefoundation

modelsparticularlysuitableforinteractingwithdynamicandmultifacetedurbanenvironments,leadingtowardsmoreinte-grated,intelligent,andresponsiveurbansystems.

UrbanFoundationModels(UFMs),asdepictedinFigure

1,

representtoanovelfamilyofmodelspre-trainedonextensive,diverseurbandatasources,encompassingmultipledatagranu-larityandmodalities.Thesemodelsexhibitadeepunderstand-ingofvariousurbandatatypesandremarkableadaptability

toawidearrayofurbantasks[7],significantlycontributing

towardstheultimaterealizationofUrbanGeneralIntelli-gence(UGI).Byintegratingandinterpretingdiverseurbandatatypes,UFMscanoffercomprehensiveinsights,uncoverintricatespatiotemporalpatterns,andenhancedecision-makingacrossvariousurbantasks.Forinstance,inthetransportationsector,UFMscananalyzeacombinationofhumaninstruc-tions,sceneimagery,trafficsensordata,andGPStrajectories

tooptimizetrafficefficiencyandsafety[8]–[10]

.Inurban

planning,UFMscancollectivelyutilizedemographic,landuse,andenvironmentaldatatoprovidecrucialinsightsfor

sustainablecitydevelopment[11],[12]

.

DespitetheburgeoninginterestandpotentialofUFMs,thefieldfacesseveralchallenges,includingtheabsenceofcleardefinitions,thelackofsystematicreviewsofexisting

1/blog/chatgpt/

2

Transportation

Urbanplanning

Energymanagement

Environmentalmonitoring

Publicsafetyandsecurity

Pre-train

Adaptation

Multi-sourceMulti-granularity

MultimodalUrbanData

UrbanFoundationModel

DownstreamApplications

Urbantextcorpus

Streetviewimages

Spatialtrajectory

Geo-sensorytimeseries

Urbanknowledgegraph

Fig.1:UrbanFoundationModels(UFMs)arepre-trainedonmulti-source,multi-granularityandmultimodalurbandataandcanbeadaptedforavarietyofdownstreamapplications.

literature,andtheneedforuniversallyapplicablesolutions[7],

[8],[13]–[16].Thispaperaddressesthesegapsbypresentinga

comprehensivesurveyonUFMs.WestartbydefiningUFMsanddiscussingtheiruniquechallenges.Buildinguponthis,weproposeadata-centrictaxonomydesignedtocategorizeandencapsulateexistingresearchonUFMs.Thistaxonomy,rootedinthediversemodalitiesandtypesofurbandata,aimstohighlighttheprogressandconcertedeffortsmadeinthisemergingdomain.Subsequently,weintroduceanewframe-workforconstructingUFMs,whichisstrategicallydevelopedtotackletheidentifiedchallengesandisversatileenoughtobegeneralizedacrossvarioustasksanddynamicurbanenvironments.Inaddition,wealsoexploretheapplicationlandscapeofUFMs,detailinghowtheycanbeappliedtoenhancedifferentaspectsofurbanintelligence.

Themajorcontributionsofthispaperareasfollows:

.WepresentthefirstcomprehensiveandsystematicreviewofUrbanFoundationModels,offeringafoundationalperspectiveonthisevolvingfield.

.ThisworkintroducestheconceptofUFMsanddelvesintothespecificchallengesassociatedwiththeirdevel-opment,sheddinglightonunexploredaspects.

.Weintroduceadata-centrictaxonomyforUFMs,whichcategorizesandclarifiesexistingresearchbasedonurbandatacharacteristics,aidingintheunderstandingofthefield’scurrentstateandfuturedirections.

.AnovelframeworkforthedevelopmentoffutureUFMsisproposed,highlightingitspotentialforgeneralizationacrossdiverseurbantasksanddynamicenvironments.

Therestofthispaperisorganizedasfollows.InSection

II,

weintroducebasicdefinitionsandconceptsofUFMsandhighlighttheuniquechallengesinvolvedinbuildingthem.Section

III

delvesintoexistingstudiesonUFMs,categorizedintolanguage-basedmodels,vision-basedmodels,trajectory-basedmodels,timeseries-basedmodels,multimodalmodels,andothers.Section

IV

presentsaprospectiveframeworkforbuildingageneralUFM.Section

V

discussesUFMs’promisingapplicationsindifferenturbandomains.Finally,weconcludethispaperinSection

VI.

II.BASICSOFURBANFOUNDATIONMODELS

A.ConceptsofUrbanFoundationModels

Definition1(UrbanFoundationModels).UrbanFoundationModels(UFMs)arelarge-scalemodelspre-trainedonvastmulti-source,multi-granularity,andmultimodalurbandata.Itbenefitssignificantlyfromitspre-trainingphase,exhibit-ingemergentcapabilitiesandremarkableadaptabilitytoabroaderrangeofmultipledownstreamtasksanddomainsinurbancontexts.

1)DataCharacteristics:UrbanFoundationModels

(UFMs)aredistinguishedbytheirabilitytoprocessandanalyzeavastarrayofdatatypes,eachcontributinguniquelytothecomprehensiveunderstandingofurbandynamics.ThedatacharacteristicsofUFMs,includingmulti-source,multi-granularity,andmulti-modalaspects,areessentialforcapturingthecomplexitiesofurbanenvironments.Thefollowingsectionsdetailthesekeydatacharacteristics.

-Multi-source.UFMsintegratedatafromdiverseurbansourcesincludingurbantext,sensornetworks,socialmedia,andsatelliteimagery,facilitatingacomprehensiveunderstand-ingofurbanenvironments.

-Multi-granularity.UFMshandledataatdifferentlevelsofgranularity.Atamacrolevel,theyprocesscity-widepatternsliketrafficfloworpopulationmovement.Atamicrolevel,theyfocusonspecificneighborhoodsorstreets,analyzinglocaltrendsandbehaviors.

-Multi-modal.UFMsexcelinintegratingdiversedatatypessuchastextual,visual,andsensor-basedinputs,enablingthemtocaptureurbancomplexitiesmoreeffectivelythansingle-data-typemodels.

2)DefinitionofDataTypes:Belowaredefinitionsand

introductionstothemajordatatypesinurbancontexts.

-Visualdata.Thisreferstoanyvisualinformationcapturedinimageorvideoformat,includingstreetviewimages,satelliteimagery,andurbansurveillancefootage.Formally,animagecanberepresentedasa3-dimensionalmatrixI(x,y,c)wherexandyrepresentthespatialdimensions(widthandheight)oftheimage,andcrepresentsthecolorchannels.Eachentryinthematrixcorrespondstoapixelvalue.Forcolorimages,

3

thisextendstoa3Dmatrixtoincludecolorchannels,typicallyRGB(Red,Green,Blue).

-Textualdata.Thisreferstodatathatiscomposedofwrittenlanguage.Examplesincludesocialmediaposts,geo-textdata,anddocumentsrelevanttourbanenvironments.Formally,textualdatacanberepresentedasasequenceoftokensT=[t1,t2,...,tn]whereeachtokenticorrespondstoawordorcharacterinthetext.Thesetokensareoftentransformedintonumericalrepresentations(likewordembeddings)forprocessinginUFMs.

-Geo-sensorydata.Thisreferstodatacollectedfromphysicalgeospatialsensorsinanurbanenvironment.ThisincludesGPSsensors,trafficsensors,weatherstations,andotherIoTde-vices.Formally,sensordatacanberepresentedasatime-seriescollectionS(t)={(s1,t1),(s2,t2),...,(sn,tn)}whereeachdatapointsiattimeticorrespondstoasensorreading.Thesereadingscanvaryinnature(e.g.,GPSpositions,trafficvolumes,andairqualityindices)dependingonthesensortype. 3)KeyTechniquesofConstructingUrbanFoundationMod-els:UFMs,similartofoundationmodelsinotherfields,primarilyrelyontwokeytechniques,i.e.,pre-trainingandadaptation.

-Pre-training.UFMstypicallyundergopre-trainingonex-tensive,varieddatasetscomprisinggeo-textdata,socialmediacontent,streetviewimages,trajectories,spatiotemporaltimeseriesdata,point-of-interests,etc.Theprimarygoalofpre-trainingistoenablethesemodelstoobtainasmuchgen-eralknowledgeandpatternsaspossible,therebycapturingtheoverarchingcharacteristicsandstructuresinherentinthedata.Pre-trainingmethodsforUFMsareprincipallydividedintofourcategories:supervisedpre-training,generativepre-

training,contrastivepre-training,andhybridpre-training[17]:

(1)Supervisedpre-training.Thismethodinvolvestrain-ingmodelsondatasetswithasubstantialvolumeofla-beleddata,pairinganinputwithitscorrespondingoutput.Thisapproachallowsmodelstolearntherelationshipbe-tweeninputsandoutputsinasupervisedmanner.SupposeD={(x1,y1),(x2,y2),...,(xN,yN)}isthetrainingdataset,wherexirepresentsthei-thsampleandyiisthecorrespondinglabel.Hereisanexampleofthemostcommonlyusedcross-

entropylossfunction[18]:

Lsup=?yiclog(ic),(1)

whereLsupisthesupervisedlossfunction,Nisthenumberofsamples,Cisthenumberofclasses,yicisabinaryindicator(0or1)ifclasslabelcisthecorrectclassificationforobservationi,icisthepredictedprobabilityobservationiisofclassc. (2)Generativepre-training.Thisapproachtrainsmodelstogenerateorreconstructinputdataintheabsenceoflabeleddata.Itnecessitatesthemodel’sabilitytoautonomouslydis-cernpatternsandregularities,therebyproducingoutputsthatcloselymimicrealdata.Hereisanexampleoflossfunctioningenerativepre-trainingcalledNegativeLog-Likelihood(NLL)

loss[19]:

Lgen=?logP(xi|x<i,θ),(2)

i=1

whereLgenisthegenerativelossfunction,Nisthenumberoftokensinthesequence,xirepresentsthei-thtoken,x<irepresentsalltokensbeforethei-thtoken,P(xi|x<i,θ)istheprobabilityassignedbythemodeltothecorrecttokenxi,giventheprecedingtokensx<iandthemodelparametersθ. (3)Contrastivepre-training.Thismethodtrainsmodelstodistinguishbetweendatainstancesandlearnrobustdatarep-resentationsbyevaluatingsimilaritiesanddifferencesamongdatasamples,allwithoutneedingexternallyprovidedlabels.

Asimpleformofthecontrastivelossforapairofsamples

canbedefinedasfollows[20]:

Lcon=yD2+(1?y)max(0,??D)2,(3)

whereLconisthecontrastiveloss,yisabinarylabelindicatingwhetherthepairhassamelabels(y=1)ordifferentlabels(y=0),Disthedistancebetweentherepresentationsofthepairofsamples,oftencalculatedusingametriclikeEuclideandistance,and?isahyperparameterthatdefinestheminimumdistancebetweendissimilarpairs.

(4)Hybridpre-training.Thisapproachintegratesmultiplepre-trainingmethods,includingsupervised,generative,andcontrastivetechniques.Thisstrategyaimstotailorthetrainingprocesstospecificproblemdemands,leveragingthestrengthsofeachmethod.

Sinceobtainingalargeamountofhigh-qualityhuman-labeleddatacanbeexpensiveorimpracticalinmanydo-mains,generativeandcontrastivepre-trainingarefrequentlyemployedtofullyleveragevarioustypesofdatasets.Thesemethodsoptimizethemodel’sperformanceandadaptability.Supervisedpre-trainingmaybeusedinspecificapplicationscenarios,particularlywithabundanthigh-qualitylabeleddata.-Adaptation.Afterpre-training,modelscanbecustomizedtospecifictasksordomainsthroughseveraladaptationmethods.TheseadaptationmethodsenhancetheflexibilityandpowerofUFMsacrossdiverseapplicationcontexts.Selectingsuitableadaptationstrategiesiscrucialformaximizingtheperformanceofpre-trainedmodelsonspecifictasksanddatasetswhilereducingresourcecosts.

(1)Modelfine-tuning.Modelfine-tuningisawidelyusedadaptationmethodinUFMs.Thisapproachinvolvesadditionaltrainingofthepre-trainedmodelusingatask-specificdataset.Theprocessoffine-tuningadjuststheparametersofthemodeltoenhanceitseffectivenessforthespecifictask.Fine-tuningisdirectandeffectivefortask-specificadaptation,butitmaydemandsignificantamountsoftrainingdataandcomputationalresources.Thismethodisparticularlyusefulwhenamplelabeleddataisavailableforthetargettask.Besides,thereare

alsosomeparameter-efficientfine-tuningmethods[21]which

minimizethenumberofparametersthatrequireadjustmentduringadaptation,maintainingabalancebetweenmodeladap-tationperformanceandresourceconsumption.

(2)Prompttuning.Divergingfromthetraditional“pre-train,fine-tune”paradigm,prompt-tuningemployslightweightprompttokenscontainingrelevantinformationaboutthetargettask.Inpractice,suchpromptsareoftenasmallnumber

oftask-specificlearnableparameters[22].Duringthefine

-tuningstage,theparametersofthepre-trainedbackbonemodelarefrozen,andonlythetrainablepromptsareadjusted.

4

Thismethodhasproveneffectiveinleveragingtheinherentknowledgeembeddedinthepre-trainedbackbone.This“Pre-

train,PromptandPredict”paradigm[23]ismoreefficient,asit

necessitatestrainingonlyaminimalsetofpromptparameters,therebyavoidingtheneedforextensivealterationstothemodel’sexistingparameters.

(3)Promptengineering.Promptengineering

[24]isa

training-freeapproachtodirectlyadoptfoundationmodelspre-trainedonothermodalities(likelargelanguagemodels)withoutalteringtheparametersofpre-trainedmodels.Itin-volvesdesigningtask-specificpromptsthatguidethemodel’sresponses.Thesepromptsactasinstructionsorhints,steeringthemodel’sattentiontowardrelevantinformationwithintheinputdata.

Theseadaptationmethods,includingfine-tuning,prompt-tuning,andpromptengineering,provideUFMswiththeadaptabilityneededtoexcelindifferenturbancontexts.Thechoiceofadaptationstrategydependsontheavailabilityofdata,computationalresources,andthespecificrequirementsofthetargettask.Eachmethoduniquelybalancesperformanceandresourceefficiency,makingUFMsversatiletoolsforurbanapplications.

B.ChallengesofBuildingUrbanFoundationModels

WhileUrbanFoundationModels(UFMs)offersignificantpotentialforunderstandingandmanagingcomplexurbanen-vironments,buildingUFMsisacomplextaskthatencountersseveralintricatechallenges.Thesechallengesstemprimarilyfromtheinherentcomplexitiesofurbandata,thedynamicnatureofurbanenvironments,andthediverserangeofappli-cationsUFMsareexpectedtoaddress.Below,wedelveintothekeychallengesthatunderscoretheintricaciesofbuildingeffectiveUFMs.

-Multi-source,multi-granularity,andmulti-modaldatainte-gration.Theintegrationofmulti-source,multi-granularity,andmulti-modaldataisoneoftheprimarychallengesinbuildingUFMs.UFMsmusteffectivelycombinedatafromvarioussourceslikesocialmedia,satelliteimagery,urbansensors,andtrafficrecords.Eachsourceprovidesdataatdifferentlevelsofgranularity,frombroadcity-widepatternstospecificlocaldetails.Moreover,thedatamodalitiesvarysignificantly,encompassingtext,images,videos,andsensorreadings.Inte-gratingthesedisparatedatatypesposessignificantchallengesindatapreprocessing,normalization,andfusion.Themodelsmustbeabletodiscernrelevantpatternsacrossthesediversedatasetsandreconcilepotentialconflictsordiscrepanciesinthedata.

-Spatio-temporalreasoningcapability.Anothermajorchal-lengeinUFMsismasteringspatio-temporalreasoning.Urbanenvironmentsaredynamic,withchangesandpatternsevolvingoverbothspaceandtime.UFMsneedtounderstandandpre-dictcomplexphenomenathataretime-dependentandspatial-dependent.Thisinvolvessophisticatedmodelingoftemporalsequencesandspatialdistributions,requiringadvancedalgo-rithmscapableofhandlinghigh-dimensionaldata.Thisspatio-temporalreasoningskillisparticularlyvitalinurbancontextsandisoftenlessemphasizedorabsentinotherfoundationmodeltypes.

-Versatilitytodiverseurbantaskdomains.Finally,theversatilityofUFMstodiverseurbantaskdomainsisasignificantchallenge.Urbanenvironmentsencompassawidearrayofdomains,suchastransportation,publicsafety,en-vironmentalmonitoring,andurbanplanning.Eachdomainpresentsuniquechallengesandrequirements.UFMsneedtobeversatileenoughtoadapttothesevariedcontexts.Thisinvolvesnotonlytailoringthemodelstospecifictasksbutalsoensuringthattheycanbegeneralizedacrossdifferenturbanscenarios.Thechallengeliesindevelopingmodelsthatarebothspecializedforparticulartasksandflexibleenoughtoaccommodatethemultifacetednatureofurbanenvironments.Thisrequiresacarefulbalancebetweenmodelcomplexityandapplicability,ensuringthatUFMscanbeeffectivelydeployedinreal-worldurbansettings.

III.OVERVIEWOFURBANFOUNDATIONMODELS

AsillustratedinFigure

2,weintroduceadata-centric

taxonomyfortheUFMs-relatedstudiestoshedlightontheprogressandeffortsmadeinthisfield.Basedontheurbandatamodalities,wecategorizetheexistingworksonUFMsintosixclasses:language-basedmodels,vision-basedmodels,trajectory-basedmodels,timeseries-basedmodels,multimodalmodels,andothers.Afterthat,weintroducethesestudiesthroughthelensoftheirfocusedpre-trainingandadaptationtechniques.AlltherelatedworksaresummarizedinTable

I

intheAppendix.

A.Language-basedModels

Currently,agreatnumberofworkshavebeendonetodevelopPre-trainedLanguageModels(PLMs).PLMstypi-callyleverageTransformermodelsasthebackboneandarepre-trainedoverlarge-scaleunlabeledcorpora,demonstratingstrongcapabilitiesinnaturallanguageprocessing(NLP)tasks.

EarlyrepresentativePFMs,suchasBERT[25],XLNet[26],

andGPT-seriesmodels[19],[24](i.e.

,GPT-1andGPT-2),adopta”pre-trainingandfine-tuning”paradigmfordown-streamtasks.Recently,researchershavefoundthatincreasingtheparameterscaleofPLMstoasignificantsize(e.g.,tensofbillionsofparameters)notonlyenhancestheircapac-itybutalsoleadstosurprisingemergentabilities(e.g.,in-contextlearning,reasoning),whicharenotpresentinsmallerPLMs(e.g.,BERT).TheseenlargedPLMsarealsotermed

LargeLanguageModels(LLMs)inliterature[27].Anotable

progressofLLMsisthereleaseofChatGPT,whichshowsexceptionalperformanceinchattingwithhumansandhasgarneredwidespreadattentionfromsociety.

Peoplelivingincitieshavegeneratedanextensivecorpusoftextualdatareflectingurbanpatterns,includingdocuments(e.g.,trafficreports),dialogs(e.g.,ride-hailingconversations),andgeo-texts(e.g.,geo-taggedtweets).Therefore,itises-sentialtoexploretheapplicationofPLMsinurbancontext.Priorresearchprimarilyfallsintotwocategories:pre-trainingandadaptationapproaches.Wewillrevieweachoftheminsubsequentsections.

5

Fig.2:Adata-centrictaxonomyforexistingUFMs-relatedworksbasedonthetypesofurbandatamodalities.

1)Pre-training:Extensiveeffortshavebeendirectedto-

wardsthedevelopmentofgeneral-purposePLMs.Thesemod-elsaretrainedontextcorporathatcoverawidearrayoftopicsanddomains.Whilespecializedurbandatasets,suchastrafficreportsandsocialmediaposts,havebeenincorporatedduringtraining,thegoalofthemistoimprovethebroadcapabilities

ofPLMs.Recently,severalstudies[28],[29]focusontraining

PLMsbyonlyusingdomain-specificdata,outperforminggeneralistcounterpartsontaskswithinthosedomains.For

example,ERNIE-GeoL[28]isapre-trainedlanguagemodel

dedicatedtoimprovinggeography-relatedtasksatBaiduMaps.Themodelispre-trainedonmassivedataextractedfromaheterogeneousgraphthatcontainsrichtoponymandspatialknowledge,byusingmaskedlanguagemodelingand

geocodingtasks.Dingetal.[29]proposeMGeo,alanguage

modelspecificallypre-trainedonabundantgeographiccontextdataforquery-POImatching.MGeodesignsageographicalencodertoencodegeographiccontextrepresentations.Italsoutilizesmaskedlanguagemodelingandcontrastivelearninginthepre-trainingstage.However,large-scalepre-trainingoftenrequiressubstantialcomputationalresourcesandelectricity,whichmaynotbeeasyforresearcherstoobtain.

2)Adaptation:Insteadoftrainingurbanlanguagemodels

fromscratch,previousstudiesmainlyfocusonadaptingexist-ingPLMstourbanscenarios.Inthisway,wecanmakefull

useoftheworldknowledgeencapsulatedinexistingPLMswhilesignificantlyreducingthedemandedcomputationalcost.Therearetwocategoriesofapproaches:promptengineeringandmodelfine-tuning.

-Promptengineering.PromptingaimstosteerPLM’sbe-haviortowarddesiredoutcomesviapre-definedtextinputs,i.e.,taskdescriptionorasetofdemonstrations,whichallowsPLMstohandletaskstheyhaveneverexplicitlybeentrainedfor.Promptingoffersaflexible,efficient,anduser-friendlywaytoharnessthecapabilitiesofPLMswithoutthenecessityofretrainingorfine-tuningthemodel.Inrecentliterature,severalpreliminarystudiessuggestthatPLMshaveencodedawealthofurbanfactualandspatio-temporalknowledgefromtheirtrainingcorpora.Theseknowledgecanbeeffectively

queriedbypromptingtechniques.Forinstance,Gurnee[30]

conductsempiricalstudiesdemonstratingthatLLMslearnrepres