2019機(jī)器學(xué)習(xí)李宏毅Transformer-(v5)

Transformer李宏毅

Hung-yiLeeBERTTransformerSeq2seqmodelwith“Self-attention”SequenceHardtoparallel!

PreviouslayerNextlayer

UsingCNNtoreplaceRNN………………SequenceHardtoparallel

PreviouslayerNextlayer

(CNNcanparallel)……

FiltersinhigherlayercanconsiderlongersequenceUsingCNNtoreplaceRNNSelf-Attention

Self-AttentionLayer

YoucantrytoreplaceanythingthathasbeendonebyRNNwithself-attention.

Self-attention

/abs/1706.03762Attentionisallyouneed.

ScaledDot-ProductAttention:Self-attention

dotproduct

拿每個queryq去對每個keyk

做attention

Self-attention

Soft-max

Self-attention

Consideringthewholesequence

Self-attention

拿每個queryq去對每個keyk

做attention

Self-attention

Self-AttentionLayer

Self-attention

=

=

=

Self-attention

Self-attention

Self-attention

=

Self-attention反正就是一堆矩陣乘法，用GPU可以加速=

=

=

=

=

Multi-headSelf-attention

(2headsasexample)

Multi-headSelf-attention

(2headsasexample)

Multi-headSelf-attention

(2headsasexample)

=

PositionalEncoding

+

100…………i-thdim

+=

sourceofimage:http://jalammar.github.io/illustrated-transformer/

+=

-11Seq2seqwithAttention

Encoder

Decoder

Review:/watch?v=ZjfjPzXw6og&feature=youtu.be

Self-AttentionLayerSelf-AttentionLayer/2017/08/transformer-novel-neural-network.htmlTransformerEncoderDecoderUsingChinesetoEnglishtranslationasexample機(jī)器學(xué)習(xí)<BOS>machinemachinelearning…Transformer

+

Masked:attendonthegeneratedsequenceattendontheinputsequence/abs/1607.06450BatchSize

/watch?v=BZh1ltr5RkgLayerNorm:BatchNorm:LayerNormLayerBatchAttentionVisualization/abs/1706.03762AttentionVisualizationTheencoderself-attentiondistributionfortheword“it”fromthe5thtothe6thlayerofaTransformertrainedonEnglishtoFrenchtranslation(oneofeightattentionheads)./2017/08/transformer-novel-neural-network.htmlMulti-headAttentionExampleApplicationIfyoucanuseseq2seq,youcan