移動(dòng)環(huán)境下的內(nèi)容推薦初探

移動(dòng)環(huán)境下的內(nèi)容推薦初探Agenda●

什么是推薦系統(tǒng)●

移動(dòng)內(nèi)容推薦的特點(diǎn)●

豌豆莢移動(dòng)內(nèi)容推薦關(guān)鍵技術(shù)●

總結(jié)什么是推薦系統(tǒng)Recommender

systems

orrecommendation

systems

(sometimesreplacing

"system"

with

a

synonym

suchas

platform

or

engine)

are

a

subclass

ofinformation

filtering

system

that

seek

topredict

the

'rating'

or

'preference'

thatuser

would

give

to

an

item.-

from

wikipediaRecommender

systems/engines

are

asubclass

of

information

retrievalsystems/engines

in

passive

mode.Search

systems/engines

are

a

subclassof

information

retrieval

systems/enginesin

active

mode.常用的推薦功能●

有哪些值得看的電影？●

去哪兒下館子腐??？●

淘點(diǎn)啥寶貝？●

裝什么有意思的應(yīng)用？●

大家都在討論什么話題？●

Netflix

Prize

$1

Million

Challenge?應(yīng)用場(chǎng)景排行榜推薦個(gè)性化推薦使用海量數(shù)據(jù)電影推薦多多是餐館推薦多少是商品推薦多多是應(yīng)用推薦多多是話題推薦多少是廣告推薦少多是推薦技術(shù)應(yīng)用場(chǎng)景推薦用戶(User)海量數(shù)據(jù)是推薦之本

推薦位置

●

推薦位所在頁(yè)面內(nèi)容

●

推薦位在頁(yè)面中位置

推薦物品(Item)推薦物品

●

種類、主題、關(guān)鍵詞

●

生命周期、投放規(guī)則

●

大小、顏色、價(jià)格

●

…...推薦用戶

●

人口學(xué)屬性(性別、年

齡等)

●

行為屬性/興趣特征(瀏

覽、搜索、緩存)

●

Social內(nèi)容(社交關(guān)系

、小組發(fā)帖)

●

上下文(時(shí)間、地點(diǎn)等)●…...

推薦算法

關(guān)鍵規(guī)則挖掘協(xié)同過(guò)濾

CF(User-based

CF,

Item-

based

CF)

矩陣分解

圖挖掘

基于熱度/知識(shí)/規(guī)則的推薦

基于內(nèi)容的推薦

基于預(yù)測(cè)模型的推薦

混合推薦方法推薦算法

推薦用戶

用戶行為屬性

Social內(nèi)容

人口學(xué)屬性

行為屬性/興趣

特征

用戶全局畫像

推薦場(chǎng)景一個(gè)用戶在多個(gè)物品

有行為

用戶-物品行為矩陣

不能過(guò)于稀疏用戶對(duì)物品的興趣規(guī)

律可觀察

適用于解決冷啟動(dòng)

(cold

start)問(wèn)題真實(shí)系統(tǒng)大都這么做推薦算法優(yōu)點(diǎn)缺點(diǎn)示例基于知識(shí)的推薦算法(基于人工觀察和經(jīng)驗(yàn)，抽象出規(guī)則進(jìn)行推薦)1.方法簡(jiǎn)單2.規(guī)則易解釋、易運(yùn)營(yíng)3.所需數(shù)據(jù)少1.規(guī)則來(lái)源于人工知識(shí)，覆蓋率差2.有效周期難以判斷好友推薦基于內(nèi)容的推薦算法1.可以解決冷啟動(dòng)問(wèn)題2.方便整合各種用戶/物品特征3.推薦結(jié)果易于理解1.受限于特征提取的準(zhǔn)確性2.需要大量的支撐數(shù)據(jù)3.常局限于特征稀疏的情況GoogleNewsAmazon淘寶豌豆莢協(xié)同過(guò)濾的推薦算法1.原理簡(jiǎn)單有效，易于理解2.算法靈活，支持海量用戶/物品3.無(wú)需考慮復(fù)雜特征，支持特征空間極其復(fù)雜的用戶/物品的高效計(jì)算1.很明顯的冷啟動(dòng)問(wèn)題2.常受限于用戶/物品行為矩陣稀疏的情況3.受限于用戶/物品rating的正確性Netfix淘寶Amazon豆瓣豌豆莢常用推薦算法比較協(xié)同過(guò)濾-User-based

CF協(xié)同過(guò)濾-Item-based

CF移動(dòng)推薦的特點(diǎn)使用次數(shù)多時(shí)間碎片化內(nèi)容消費(fèi)占大頭用戶易定位(用戶<->設(shè)備)特點(diǎn)原因?qū)崟r(shí)推薦非常重要移動(dòng)設(shè)備/應(yīng)用使用次數(shù)多推薦的準(zhǔn)確率要求更高移動(dòng)設(shè)備/應(yīng)用使用時(shí)間碎片化內(nèi)容推薦是關(guān)鍵移動(dòng)設(shè)備/應(yīng)用消費(fèi)中內(nèi)容是大頭可利用用戶信息更豐富移動(dòng)設(shè)備可以用來(lái)準(zhǔn)確定位用戶移動(dòng)推薦的新特點(diǎn)豌豆莢移動(dòng)內(nèi)容推薦關(guān)鍵技術(shù)豌豆莢移動(dòng)內(nèi)容推薦框架關(guān)鍵技術(shù)之知識(shí)圖譜知識(shí)圖譜

-

Knowledge

Graph

Knowledge

Graph

understands

real-world

entities

and

their

relationships

to

one

another:

things,

not

strings,

helps

the

machine

to

understand

the

world

a

bit

more

like

people

do.Knowledge

Graph

consists

of

nodes

with

edgesbetween

them.

Each

node

corresponds

to

an

entity

inreal

work,

each

edge

describe

the

relationship

betweenthe

begin

and

end

nodes.Google

Knowledge

GraphData

Sources●

Public

Sources(Wikipedia,

Freebase,CIA

World

Factbook)●

Knowledge

Mining

from

Webpages(Lists,

Tables,

etc.)●

Knowledge

Mining

from

user

queries(xxattributes

of

yy

entity)Google

Knowledge

Graph

Usages

●

Find

the

right

thing

○

Recognize

the

entities

in

queries

and

associates

them

with

the

relevant

information,

and

disambiguate

entities

using

context

information.

●

Get

the

best

summary

○

Use

the

users

search

behavior

to

determine

the

most

needed

aspects

for

each

entity.

●

Go

deeper

and

broader○Help

users

to

get

some

unexpected

discoveries.豌豆莢知識(shí)圖譜數(shù)據(jù)來(lái)源●

公開(kāi)來(lái)源(Wikipedia,

Freebase,

IMDB、豆瓣電影)●

應(yīng)用內(nèi)內(nèi)容/網(wǎng)頁(yè)信息中解析出來(lái)的實(shí)體

和實(shí)體關(guān)系信息(應(yīng)用，游戲，書，視頻

，音樂(lè)等)●

豌豆莢用戶貢獻(xiàn)的User

profile信息和UGC內(nèi)容●

用戶-實(shí)體的關(guān)系(搜索、瀏覽、消費(fèi)、豌豆莢知識(shí)圖譜數(shù)據(jù)來(lái)源●

公開(kāi)來(lái)源(Wikipedia,

Freebase,

IMDB、豆瓣電影)●

應(yīng)用內(nèi)內(nèi)容/網(wǎng)頁(yè)信息中解析出來(lái)的實(shí)體

和實(shí)體關(guān)系信息(應(yīng)用，游戲，書，視頻

，音樂(lè)等)●

豌豆莢用戶貢獻(xiàn)的User

profile信息和UGC內(nèi)容●

用戶-實(shí)體的關(guān)系(搜索、瀏覽、消費(fèi)、豌豆莢知識(shí)圖譜

用途

●

信息展示○全面展示內(nèi)容相關(guān)的實(shí)體及相互關(guān)系●

用戶意圖理解○○重名區(qū)分查詢意圖識(shí)別●

內(nèi)容推薦○○○○豌豆莢首頁(yè)內(nèi)容推薦

-

追新有趣豌豆莢詳情頁(yè)內(nèi)容推薦

-

相關(guān)而非相似豌豆莢各頻道列表頁(yè)內(nèi)容推薦

-

追熱有趣…...關(guān)鍵技術(shù)之Giraph-圖挖掘平臺(tái)Giraph

-

圖挖掘平臺(tái)Giraph

是Google

Pregel的開(kāi)源實(shí)現(xiàn)，它基于Hadoop，通過(guò)在Hadoop上運(yùn)行map-only的job來(lái)實(shí)現(xiàn)常見(jiàn)的圖挖掘算法。Giraph

提供了一個(gè)簡(jiǎn)單的“面向節(jié)點(diǎn)”(Think

like

a

vertex)的編程模型。所有的圖挖掘算法都可簡(jiǎn)化為在每一個(gè)Vertex上的兩類操作：1)

在vertext上進(jìn)行數(shù)值計(jì)算(computation)；2)

將數(shù)值計(jì)算的結(jié)果通過(guò)邊傳遞給相鄰的全部/部分節(jié)點(diǎn)(communication)。上述編程模型極大簡(jiǎn)化了圖挖掘算法的實(shí)現(xiàn)，使得每個(gè)算法都可以迭代地實(shí)現(xiàn)，這一切是建立在Giraph內(nèi)嵌的bulksynchronous

parallel編程模型基礎(chǔ)上。Giraph

Timeline●

受Google

Pregel啟發(fā)開(kāi)發(fā)

(2010)●

被Yahoo!捐獻(xiàn)給ASF

(2011)●

成為Top-level

Apache

Project

(2012)●

發(fā)布1.0版本

(2013)●

發(fā)布1.1版本

(2014)Giraph典型應(yīng)用●

Ranking

○

Popularity,

Importance,

etc.●

Label

Propagation

○

Location,

School,

Interest,

gender,

etc.●

Community

○

Groups,

Clusters

Map-only

Map-Reduce

●

Think

like

a

key-value

pairpublic

class

Mapper<

KEYIN

VALUEIN

KEYOUT

VALUEOUT>{

void

map(KEYIN

key,

VALUEIN

value,

Context

context)

throws

IOException,

InterruptedException;}

Giraph

●

Think

like

a

vertexpublic

class

Vertext<

I

extends

WritableComparable,

V

extends

Writable,

E

extends

Writable,

M

extends

Writable>{

void

compute(

Iterator<M>

msgIterator);}

Giraph

API(當(dāng)前Vertex上

可用）I

getVertextId()V

getVertextValue()void

setVertextValue(V

vertextValue)Iterator<I>

iterator()E

getEdgeValue(I

targetVertextId)boolean

hasEdge(I

targetVertextId)boolean

addEdge(I

targetVertextId,

EedgeValue)E

removeEdge(I

targetVertextId)void

voteToHalt()boolean

isHalted()Giraph

API(Vertex間傳遞消息可用)

void

sendMsg(I

targetVertexid,

M

msg)

void

sendMsgToAllEdges(M

msg)

void

addVertextRequest(BasicVertex(I,

V,

E,

M>

vertex)

void

removeVertextRequest(I

vertexId)

void

addEdgeRequest(I

sourceVertexId,

Edge<I,

E>

edge)

void

removeEdgeRequest(I

sourceVertextId,

I

destVertextId)一個(gè)簡(jiǎn)單的例子

-

計(jì)算最大節(jié)點(diǎn)值public

class

MaxValueVertex

extends

EdgeListVertex<IntWritable,

IntWritable,

IntWritable,

IntWritable>{@overridepublic

void

compute(Iterator<IntWritable>

msgIterator){boolean

changed

=

false;while(msgIterator.hasNext()){IntWritable

msgValue

=

msgIterator.next();if(msgValue.get()

>

getVertexValue().get())}setVertexValue(msgValue);changed

=

true;

}}if(getSuperStep()

==

0

||

changed)

{sendMsgToAllEdges(getVertexValue());}

else

{voteToHalt();

}

}}復(fù)雜點(diǎn)的例子

-

PageRankpublic

class

SimplePageRankVertex

extends

EdgeListVertex<

LongWritable,

DoubleWritable,

FloatWritable,

DoubleWritable>{

@override

public

void

compute(Iterator<DoubleWritable>

msgIterator){

if(getSuperStep()

>=

1)

{

double

sum

=

0;

while(msgIterator.hasNext(