關于大規(guī)模語言模型在科學研究中的應用綜述

LLM4SR:ASurveyonLargeLanguageModelsforScientific

Research

ZIMINGLUO?

,UniversityofTexasatDallas,USA

arXiv:2501.04306v1[cs.CL]8Jan2025

ZONGLINYANG?

,NanyangTechnologicalUniversity,Singapore

ZEXINXU,

UniversityofTexasatDallas,USA

WEIYANG,

UniversityofTexasatDallas,USA

XINYADU,

UniversityofTexasatDallas,USA

Inrecentyears,therapidadvancementofLargeLanguageModels(LLMs)hastransformedthelandscapeofscientificresearch,offeringunprecedentedsupportacrossvariousstagesoftheresearchcycle.ThispaperpresentsthefirstsystematicsurveydedicatedtoexploringhowLLMsarerevolutionizingthescientificresearchprocess.WeanalyzetheuniquerolesLLMsplayacrossfourcriticalstagesofresearch:hypothesisdiscovery,experimentplanningandimplementation,scientificwriting,andpeerreviewing.Ourreviewcomprehensivelyshowcasesthetask-specificmethodologiesandevaluationbenchmarks.Byidentifyingcurrentchallengesandproposingfutureresearchdirections,thissurveynotonlyhighlightsthetransformativepotentialofLLMs,butalsoaimstoinspireandguideresearchersandpractitionersinleveragingLLMstoadvancescientificinquiry.Resourcesareavailableatthefollowingrepository:

/du-nlp-lab/LLM4SR.

CCSConcepts:?Computingmethodologies→Naturallanguageprocessing;?Generalandreference→Surveysandoverviews.

AdditionalKeyWordsandPhrases:LargeLanguageModels,ScientificHypothesisDiscovery,ExperimentPlanningandImplementation,AutomatedScientificWriting,PeerReviewGeneration

ACMReferenceFormat:

ZimingLuo,ZonglinYang,ZexinXu,WeiYang,andXinyaDu.2025.LLM4SR:ASurveyonLargeLanguageModelsforScientificResearch.ACMComput.Surv.1,1(January2025),

37

pages.

/10.1145/

nnnnnnn.nnnnnnn

Automating

Research

Process§3.3

Draftingand

Writing

§4.4

CitationText

Generation

§4.2

RelatedWork

Generation

§4.3

Scientific

Hypothesis

Discovery

§2

Peer

Reviewing§5

OptimizingExperimentDesign§3.2

PaperWriting§4

ExperimentPlanning&Implementation

§3

Fig.1.Schematicoverviewofthescientificresearchpipelinecoveredinthissurvey.Thiscyclicalprocessbeginswithscientifichypothesisdiscovery,followedbyexperimentplanningandimplementation,paperwriting,andfinallypeerreviewingofpapers.Theexperimentplanningstageconsistsofoptimizingexperimentdesignandexecutingresearchtasks,whilethepaperwritingstageconsistsofcitationtextgeneration,relatedworkgeneration,anddrafting&writing.

*Bothauthorscontributedequallytothiswork.

Authors’ContactInformation:

ZimingLuo

,ziming.luo@,UniversityofTexasatDallas,Dallas,Texas,USA;

ZonglinYang

,zonglin001@.sg,NanyangTechnologicalUniversity,Singapore,Singapore;

ZexinXu,

zexin.xu@,UniversityofTexasatDallas,Dallas,Texas,USA;

WeiYang,

wei.yang@,UniversityofTexasatDallas,Dallas,Texas,USA;

XinyaDu

,xinya.du@,UniversityofTexasatDallas,Dallas,Texas,USA.

2025.ACM1557-7341/2025/1-ART

/10.1145/nnnnnnn.nnnnnnn

Preprint.

2LuoandYangetal.

1Introduction

“IfIhaveseenfurther,itisbystandingontheshouldersofgiants.”

—IsaacNewton

ThescientificresearchpipelineisatestamenttotheachievementsoftheEnlightenmentinsystematicinquiry

[17,

58,

58

].Inthistraditionalparadigm,scientificresearchinvolvesaseriesofwell-definedsteps:researchersstartbygatheringbackgroundknowledge,proposehypotheses,designandexecuteexperiments,collectandanalyzedata,andfinallyreportfindingsthroughamanuscriptthatundergoespeerreview.Thiscyclicalprocesshasledtogroundbreakingadvance-mentsinmodernscienceandtechnology,yetitremainsconstrainedbythecreativity,expertise,andfinitetimeandresourcesavailableinherenttohumanresearchers.

Fordecades,thescientificcommunityhassoughttoenhancethisprocessbyautomatingaspectsofscientificresearch,aimingtoincreasetheproductivityofscientists.Earlycomputer-assistedresearchcandatebacktothe1970s,introducingsystemssuchasAutomatedMathematician

[74,

75

]andBACON

[71

],whichshowedthepotentialofmachinestoassistinspecializedresearchtasksliketheoremgenerationandempiricallawidentification.Morerecently,systemssuchasAlphaFold

[62

]andOpenFold

[4

]haveexemplifiedpioneeringeffortstoautomatespecificresearchtasks,significantlyspeedingupscientificprogressintheirrespectivedomainsbythousandsoftimes.YetitwasonlywiththeadventoffoundationmodelsandtherecentexplosioninLargeLanguageModels(LLMs)

[2,

154

]thatthevisionofcomprehensiveAIassistanceacrossmultipleresearchdomainsbecamerealistic

[190

].

TherecentyearshavewitnessedremarkableadvancementsinLLMs,transformingvariousfieldsofAIandNaturalLanguageProcessing(NLP).Thesemodels,suchasGPT-4

[2

]andLLaMA

[154

],havesetnewbenchmarksinunderstanding,generatingandinteractingwithhumanlanguage.Theircapabilities,enhancedbymassivedatasetsandinnovativearchitectures,nowextendbeyondconventionalNLPtaskstomorecomplexanddomain-specificchallenges.Inparticular,theabilityofLLMstoprocessmassiveamountsofdata,generatehuman-liketext,andassistincomplexdecision-makinghascapturedsignificantattentioninthescientificcommunity

[92,

141

].ThesebreakthroughssuggestthatLLMshavethepotentialtorevolutionizethewayscientificresearchisconducted,documented,andevaluated

[156,

165,

174

].

Inthissurvey,weexplorehowLLMsarecurrentlybeingappliedacrossvariousstagesofthescientificresearchprocess.Specifically,weidentifyfourgeneraltaskswhereLLMshavedemonstratednotablepotential.Webeginbyexploringtheirapplicationinscientifichypothesisdiscovery,whereLLMsleverageexistingknowledgeandexperimentalobservationstosuggestnovelresearchideas.Thisisfollowedbyareviewoftheircontributionstoexperimentplanningandimplementation,whereLLMsaidinoptimizingexperimentaldesign,automatingworkflows,andanalyzingdata.Wealsocovertheiruseinscientificwriting,includingthegenerationofcitations,relatedworksections,andevendraftingentirepapers.Finally,wediscusstheirpotentialinpeerreview,whereLLMssupporttheevaluationofscientificpapersbyofferingautomatedreviewsandidentifyingerrorsorinconsistencies.Foreachofthesetasks,weprovideacomprehensivereviewofthemethodologies,benchmarks,andevaluationmethods.Moreover,thesurveyidentifiesthelimitationsofeachtaskandhighlightsareasneedingimprovement.ByanalyzingthevariousstagesoftheresearchcyclewhereLLMscontribute,thissurveycaninspireresearcherstoexploreemergingconcepts,developevaluationmetrics,anddesigninnovativeapproachestointegrateLLMsintotheirworkflowseffectively.

ComparisonwithExistingSurveys.ThissurveyprovidesabroaderandmorecomprehensiveperspectiveontheapplicationsofLLMsacrosstheentirescientificresearchcyclecomparedtopriorspecializedstudies.Forexample,Zhangetal.

[187

]reviewover260LLMsinscientificdiscovery

Preprint.

Preprint.

LLM4SR:ASurveyonLargeLanguageModelsforScientificResearch3

Literature-based

Discovery(§2.2.1)

InductiveReasoning(§2.2.2)

MainTrajectory(§2.3.1)

OtherMethods(§2.3.2)

History(§

2.2)

LBD[47,151],DBLP[155],LinkPredictionModels[152,160,171]

Norton[113],Yangetal.[175],Yangetal.[173],Zhongetal.[191],Zhuetal.[194],Wangetal.[163],Qiuetal.[120]

SciMON[159],MOOSE[174],MCR[145],Qi[119],

FunSearch[130],ChemReasoner[146],HypoGeniC[193],

Scientific

HypothesisDiscovery(§

2)

ResearchAgent[9],LLM-SR[140],SGA[105],AIScientist[103],MLR-Copilot[84],IGA[141],SciAgents[41],Scideator[121],

DevelopmentofMethods(§2.3)

MOOSE-Chem[176],VirSci[148],CoI[77],Nova[49],CycleResearcher[167],SciPIP[164]

Socraticreasoning[30],IdeaSynth[118],HypoRefine[96],LDC[80]

DiscoveryBench[108],DiscoveryWorld[57]

SciMON[159],Tomato[174],Qietal.[119],Kumaretal.[68],Tomato-Chem[176] Benchmarks(§2.4)

Evaluation(§2.5)

LLM-based/Expert-basedEvaluation;DirectEvaluation/Reference-basedEvaluation;

DirectEvaluation/Comparison-basedEvaluation;RealExperimentEvaluation

HuggingGPT[136],CRISPR-GPT[52],ChemCrow[15],Coscientist[14],LLM-RDF[131],AutoGen[168],Lietal.[81],Lietal.[90]

OptimizingExperi-

LargeLanguageModels(LLMs)forScientificResearch

mentalDesign(§

3.2)

DataPreparation

(§3.3.1)

ExperimentPlanning

andImple-mentation(§

3)

Clearning[21,185],Labeling[153],FeatureEngineering[46],Synthesis[82,85,98]

ExperimentExecution

——andWorkflow

Automation(§3.3.2)

DataAnalysisand

ChemCrow[15],Coscientist[14],Wangetal.[157],Ramosetal.[124],ChatDrug[99],DrugAssist[179],ESM-1b[128],ESM-2[95],

FerruzandH?cker[35],Heetal.[44]

AutomatingExperi-mentalProcess(§3.3)

Singhetal.[143],Lietal.[79],MentalLLaMA[172],

Interpretation(§3.3.3)

Daietal.[27],Rasheedetal.[126],Zhaoetal.[188],Oliveretal.[114]

TaskBench[137],DiscoveryWorld[57],MLAgentBench[54],AgentBench[100],Spider2-V[16], DSBench[61],DS-1000[70],CORE-Bench[142],SUPER[13],MLE-Bench[20],LAB-Bench[72],

ScienceAgentBench[24]

Xingetal.[170],AutoCite[161],BACO[40],GuandHahnloser[43],Jungetal.[63]

Benchmarks&

Evaluation(§3.4)

CitationText

Generation(§4.2)

Zimmermannetal.[197],Agarwaletal.[3],Huetal.[50],Shietal.[138],Yuetal.[181],Susnjaketal.[150],LitLLM[3],HiReview[50],Nishimuraetal.[112]

RelatedWork

PaperWriting(§

4)

Generation(§4.3)

Augustetal.[8],SCICAP[48],PaperRobot[160],Ifarganetal.[56],CoAuthor[73],AutoSurvey[165],AIScientist[103]

ALCE[38],CiteBench[37],SciGen[111],SciXGen[22]

DraftingandWriting(§4.4)

Benchmarks&

Evaluation(§4.5)

ReviewRobot

[162]

,Reviewer2

[39]

,SWIF2T

[18]

,SEA

[180]

,MARG

[28]

,MetaGen

[11]

,Kumaretal.

[67]

,MReD

[135]

,CGI2

[184]

,CycleReviewer

[167]

AutomatedPeer

ReviewingGeneration(§

5.2)

PaperMage[101],CocoSciSum[29]

ReviewerGPT[97],PaperQA2[144],Scideator[122]

ReviewFlow[149],CARE[198],DocPilot[110]

Information

Summarization

PeerReview-ing(§

5)

ErrorDetection&

QualityVerification

LLM-assistedPeerReviewWorkflows(§

5.3)

Benchmarks&

Evaluation(§

5.4)

ReviewWritingSupport

MOPRD

[94]

,ORSUM

[184]

,MReD

[135]

,PeerSum

[78]

,NLPeer

[33]

,PeerRead

[65]

,ASAP-Review

[183]

,ReviewCritiqe

[32]

,Reviewer2

[39]

Fig.2.Themaincontentflowandcategorizationofthissurvey.

acrossvariousdisciplines,focusingprimarilyontechnicalaspectssuchasmodelarchitecturesanddatasets,withoutsituatingtheirroleswithinthebroadercontextoftheresearchprocess.Similarly,othersurveystendtoadoptnarrowerscopes,examiningspecificcapabilitiesofLLMsforgeneralapplications,suchasplanning

[55

]orautomation

[158

],ratherthantheirfocusedutilityinscientificresearchworkflows.Additionally,someworksaddressgeneralapproachesrelevanttospecificresearchstagesbutarenotexclusivelycenteredonLLMs,suchasrelatedworkandcitationtext

4LuoandYangetal.

Preprint.

generation

[89

]orpeerreviewprocesses

[33

].Incontrast,thissurveyintegratesthesefragmentedperspectives,providingaholisticanalysisofLLMs’contributionsacrossthescientificworkflowandhighlightingtheirpotentialtoaddressthediverseandevolvingdemandsofmodernresearch.

OrganizationofthisSurvey.AsillustratedinFigure

2

,thestructureofthissurveyisasfollows:§

2

coversLLMsforscientifichypothesisdiscovery,includinganoverviewofmethodologiesandkeychallenges.§

3

focusesonexperimentplanningandimplementation,highlightinghowLLMscanoptimizeandautomatetheseprocesses.§

4

delvesintoautomatedpaperwriting,includingcitationandrelatedworkgeneration,while§

5

exploresLLM-assistedpeerreview.Foreachtopic,thesurveyconcludeswithasummaryofcurrentchallengesandfuturedirectionsinthisrapidlyevolvingfield.

2LLMsforScientificHypothesisDiscovery

2.1Overview

Beforetheemergenceofthefield“LLMsforscientifichypothesisdiscovery”,themostrelatedpreviousresearchdomainsare“l(fā)iterature-baseddiscovery”and“inductivereasoning”.Wefirstsummarizetheresearchinthetworelateddomains(ashistory),thensummarizethemethods,benchmarks,evaluationdevelopmenttrends,andimportantprogress,andfinallyconcludewiththemainchallengesinthediscoverytask.

2.2HistoryofScientificDiscovery

UsingLLMstogeneratenovelscientifichypothesesisanewresearchtopic,mostlyoriginatingfromtworelatedresearchdomains,whichare“l(fā)iterature-baseddiscovery”and“inductivereasoning”.

2.2.1Literature-basedDiscovery.Literature-baseddiscovery(LBD)wasfirstproposedbySwanson

[151

].Thecentralideaisthat“knowledgecanbepublic,yetundiscovered,ifindependentlycreatedfragmentsarelogicallyrelatedbutneverretrieved,broughttogether,andinterpreted.”Therefore,howtoretrievepublicknowledgethatcanbebroughttogethertocreatenewknowledgeremainsachallenge.

Swanson

[151

]proposeaclassicformalizationofLBD,whichisthe“ABC”modelwheretwoconceptsAandCarehypothesizedaslinkediftheybothco-occurwithsomeintermediateconceptBinpapers.Morerecentworkhasusedwordvectors

[155

]orlinkpredictionmodels

[152,

160,

171]

todiscoverlinksbetweenconceptstocomposehypotheses.

However,classicLBDmethodsdonotmodelcontextsthathumanscientistsconsiderintheideationprocess,andarelimitedtopredictingpairwiserelationsbetweendiscreteconcepts

[47

].Toovercometheselimitations,Wangetal.

[159

]makethefirstattempttogroundLBDinanaturallanguagecontexttoconstrainthegenerationspace,andalsousegeneratedsentencesasoutputinsteadofonlypredictingrelationsasinthetraditionalLBD.

AnotherlimitationofLBDisthatithaslongbeenthoughtofasonlybeapplicabletoaveryspecific,narrowtypeofhypothesis

[159]

.However,recentprogressinscientificdiscoveryindicatesthatLBDmighthaveamuchwiderapplicablescope.Particularly,Yangetal.

[174

]andYangetal.

[176

]discussextensivelywithsocialscienceandchemistryresearcherscorrespondingly,andfindthatmostexistingsocialscienceandchemistrypublishedhypotheses(insteadofonlyanarrowtypeofhypotheses)canbeformulatedinaLBDpattern.Itprobablyindicatesthatfuturehypothesesinsocialscienceandchemistrytobepublishedcanalsoresultfrom(correct)linkagesandassociationsofexistingknowledge.

2.2.2InductiveReasoning.Inductivereasoningisaboutfindingageneral“rule”or“hypothesis”thathasawideapplicationscopefromspecific“observations”

[175

].Forexample,Geocentrism,

LLM4SR:ASurveyonLargeLanguageModelsforScientificResearch5

Preprint.

Heliocentricism,andNewton’sLawofGravityareallproposed“rules”basedonthe“observations”ofthemovementsofstarsandplanets.Scientificdiscoveryisadifficulttaskofinductivereasoningtoanextreme,whereeach“rule”isanovelscientificfinding.

Thephilosophyofsciencecommunityhassummarizedthreefundamentalrequirementsfora“rule”frominductivereasoning

[113

],whichare(1)“rule”shouldnotbeinconflictwith“observa-tions”;(2)“rule”shouldreflectthereality;(3)“rule”shouldpresentageneralpatternthatcanbeappliedtoalargerscopethanthe“specific”observations,coveringnewinformationnotexistingintheobservations.Previouslyinductivereasoningresearchismainlyconductedbythe“inductive logicprogramming”community

[26

],whichusesformallanguageandsymbolicreasoners.Yangetal.

[173

]firstworkongenerativeinductivereasoningintheNLPdomain,whichistogeneratenaturallanguagerulesfromspecificnaturallanguageobservationswithlanguagemodels,introduc- ingtherequirementsoninductivereasoningfromthephilosophyofsciencecommunity.Motivatedbytheempiricalexperiencethatlanguagemodelstendtogeneratevagueandnotspecificrules,theyadditionallyproposethefourthrequirement:(4)“rule”shouldbeclearandinenoughdetail.Thefourthrequirementmighthavebeenoverlookedbythephilosophyofsciencecommunitysince it’stooobvious.Motivatedbytherequirements,Yangetal.

[173

]designanoverly-generation-then-filteringmechanism,leveraginglanguagemodelstofirstgeneratemanypreliminaryrulesandthenfilterthosedonotsatisfytherequirements.Thenmethodsaredevelopedtouseself-refinetoreplacefilteringandusemorereasoningstepsforbetterrules

[120,

163,

191,

194

].However,the“rules”thislineofworkstrytoinduceareeitherknownknowledge,ornotscientificknowledgebutsynthesizedpatterns.

Yangetal.

[174

]makethefirstattempttoextendtheclassicinductivereasoningtasksetting(todiscoverknown/syntheticknowledge)intoarealscientificdiscoverysetting:toleverageLLMstoautonomouslydiscovernovelandvalidsocialsciencescientifichypothesesfromthepubliclyavailablewebdata.Specifically,theycollectnews,businessreviews,andWikipediapagesonsocialscienceconceptsasthewebdatatodiscoverhypothesis.

Majumderetal.

[107,

108

]furtherproposetheconceptof“data-drivendiscovery”,whichistodiscoverhypothesesacrossdisciplineswithallthepublicexperimentaldataontheweb(andprivateexperimentaldataathand).Theirmotivationisthatthepotentialofthelargeamountofpubliclyavailableexperimentaldatahasnotbeenfullyexploitedthatlotsofnovelscientifichypothesescouldbediscoveredfromtheexistingdata.

2.3DevelopmentofMethods

Amongthemethodsdevelopedforscientificdiscovery,thereisoneclearmethoddevelopmenttrajectory.Webeginbyintroducingthistrajectory,followedbyanexplorationofothermethods.

2.3.1MainTrajectory.Ingeneral,thismethoddevelopmenttrajectoryforscientificdiscoverycanbeseenasincorporatingmorekeycomponentsintothemethods.Table

1

summarizesthekeycomponentsweidentifyasimportantandindicateswhethereachmethodincorporatesthem.Specifically,theyare“strategyofinspirationretrieval”,“noveltychecker”,“validitychecker”,“claritychecker”,“evolutionaryalgorithm”,“l(fā)everageofmultipleinspiration”,“rankingofhypothesis”,and“automaticresearchquestionconstruction”.Here,each“keycomponent”referstoadetailedanduniquemethodologythathasproveneffectiveforscientificdiscoverytasks.Weexcludebroadgeneralconceptsthatmayintuitivelyseemhelpfulbutit’snotclearhowaspecificmethodfromtheconceptcanbeeffectiveforthistask(e.g.,toolusage).Next,weintroducethesekeycomponents.Foreachkeycomponent,weuseoneortwoparagraphstogiveashortoverview,summarizingitsdevelopmenttrace.ThereferenceinformationforeachmethodmentionedinthissectioncanbefoundinTable

1.

6LuoandYangetal.

Preprint.

InspirationRetrievalStrategy.Inadditiontorelyingonbackgroundknowledge,literature-baseddiscovery(LBD)facilitatestheretrievalofadditionalknowledgeasasourceofinspirationforformulatingnewhypotheses.SciMON

[159

]firstintroducestheconceptsofLBDtothediscoverytask,demonstratingthatnewknowledgecanbecomposedoflinkageofexistingknowledge.Itisvitalthattheinspirationshouldnotbeknowntoberelatedtothebackgroundbefore,oratleastshouldnotbeusedtoassociatewiththebackgroundinaknownway

[176

].Otherwise,thehypothesiswouldnotbenovel.

Inspiredbythe“ABC”modelinclassicLBDformalization,givenabackgroundknowledge,SciMONretrievessemanticallysimilarknowledge,knowledgegraphneighbors,andcitationgraphneighborsasinspirations.Specifically,twoknowledgeareidentifiedas“semanticallysimilar”iftheirembeddingsfromSentenceBERT

[127

]havehighcosinesimilarity;Theknowledgegraphtheybuiltfollowsa“[method,used-for,task]”format.ResearchAgentstrictlyfollowsthe“ABC”modelbyconstructingaconceptgraph,wherealinkrepresentsthetwoconnectedconceptnodeshaveappearedinthesamepaperbefore.Itretrievesinspirationconceptsthatareconnectedwiththebackgroundconceptsontheconceptgraph(conceptco-occurence).Scideatorretrievesinspirationpapersbasedonsemanticmatching(semanticscholarAPIrecommendations)andconceptmatch-ing(paperscontainingsimilarconceptsinthesametopic,samesubarea,anddifferentsubarea).SciPIP

[164

]retrievesinspirationsfromsemanticallysimilarknowledge(basedonSentenceBERT),conceptco-occurence,andcitationgraphneigbors.Itproposesfilteringmethodstofilternotusefulconceptsforconceptco-occurenceretrieval.

Differentfromselectingsemanticorcitationneighborsasinspirations,SciAgentsrandomlysampleanotherconceptthatisconnectedwiththebackgroundconceptinacitationgraph(viaalongorshortpath)astheinspiration.

MOOSE

[174

]proposestouseLLM-selectedinspirations:giventheresearchbackgroundandsomeinspirationcandidatesinthecontext,andaskanLLMtoselectinspirationsfortheresearchbackgroundfromthecandidates.ThenMOOSE-Chem

[176

]alsoadoptsit.MOOSE-Chemassumesthataftertrainingonhundredsofmillionsofscientificpapers,themostadvancedLLMsmightalreadyhaveacertainlevelofabilitytoidentifytheinspirationknowledgeforthebackgroundtocomposeanoveldiscoveryofknowledge.MOOSE-Chemanalyzesthisassumptionbyannotating51chemistrypaperspublishedin2024(whichareonlyavailableonlinein2024)withtheirbackground,inspirations,andhypothesis,andseewhetherLLMswithtrainingdataupto2023canretrievetheannotatedinspirationsgivenonlythebackground.Theirresultsshowaveryhighretrievalrate,indicatingthattheassumptioncouldbelargelycorrect.ThenNovaalsoadoptsLLM-selectedinspirations,withthemotivationthatleveragingtheLLM’sinternalknowledgetodetermineusefulknowledgefornewideasshouldbeabletosurpasstraditionalentityorkeyword-basedretrievalmethods.

FeedbackModules.Thenextkeycomponentistheiterativefeedbackonthegeneratedhypothesesintheaspectsofnovelty,validity,andclarity.ThesethreefeedbacksarefirstproposedbyMOOSE,motivatedbytherequirementsforahypothesisininductivereasoning

[113,

173

].Thesethreeaspectsareobjectiveenoughtogivefeedback,andeachofthemisessentialforagoodhypothesis.

?NoveltyChecker.Thegeneratedhypothesesshouldbeanovelfindingcomparedtotheexistingliterature.Whenahypothesistendstobesimilartoanexistinghypothesis,feedbackonenhancingitsnoveltycouldbebeneficialforhypothesisformulation.ExistingmethodsfornoveltyfeedbackareallbasedonLLMs.Ingeneral,therearethreewaystoprovidenoveltyfeedback.Thefirstmethodevaluateseachgeneratedhypothesisagainstarelatedsurvey(MOOSE);theseconditerativelyretrievesrelevantpapersforcomparison(SciMON,

LLM4SR:ASurveyonLargeLanguageModelsforScientificResearch7

Preprint.

Table1.DiscoveryMethods.Here“NF”=NoveltyFeedback,“VF”=ValidityFeedback,and“CF”=ClarityFeedback,“EA”=EvolutionaryAlgorithm,“LMI”=LeveragingMultipleInspirations,“R”=Ranking,“AQC”=AutomaticResearchQuestionConstruction.Theorderofmethodsreflecttheirfirstappearancetime.

Methods

InspirationRetrievalStrategyNFVFCFEALMIRAQC

SciMON

[159]

MOOSE

[174]

MCR

[145]

Qi

[119]

FunSearch

[130]

ChemReasoner

[146]

HypoGeniC

[193]

ResearchAgent

[9]

LLM-SR

[140]

SGA

[105]

AIScientist

[103]

MLR-Copilot

[84]

IGA

[141]

SciAgents

[41]

Scideator

[121]

MOOSE-Chem

[176]

VirSci

[148]

CoI

[77]

Nova

[49]

CycleResearcher

[167]

SciPIP

[164]

-

-

Semantic&Concept&CitationNeighbors√

√

√

LLMSelection√

-

√

√

√

√

√

√

√

√

√

-

-√

-

-

-

√

ConceptCo-occurrenceNeighbors√

-

-

-

-√

-

-

--

-

-

--

√

-

RandomSelection√

-

-

√

√

√

√

Semantic&ConceptMatching√

-

-

LLMselection√-√-√

-

-

LLMselection-

-

-

--

-

-

Semantic&Concept&CitationNeighbors-

-

-

-

-

-

-

-

-

√

-

-

-

√

-

√

-

-

√

-

-

√

-

-

-

√

√

√

-

√

-

-

-

√

-

-

-

-

-

√

-

-

-

-

-

-

√

√

√

-

-

-

-

-

-

-

√

-

-

-

√

-

-

-

-

√

-

-

-

-

-

-

-

-

√

√

-

-

-

-

√

√

-

-

-

SciAgents,Scideator,CoI);thethirddirectlyleveragestheinternalknowledgeofLLMsforevaluation(Qi,ResearchAgent,AIScientist,MOOSE-Chem,VirSci).

?ValidityChecker.Thegeneratedhypothesesshouldbevalidscience/engineeringfindingsthatpreciselyreflecttheobjectiveuniverse

[113]

.Arealvalidityfeedbackshouldbefromtheresultsofexperiments.However,itistime-consumingandcostlytoconductexperimentsforeachgeneratedhypothesis.Therefore,currently,validityfeedbackalmostentirelyreliesontheheuristicsofLLMsorothertrainedneuralmodels.TheexceptionsareFunSearch,HypoGeniC,LLM-SR,andSGA.Specifically,FunSearchisaboutgenerat