互联网-浅谈DeepSeek的创新性，对比科技巨头和OpenAI

互联网入华泰证券HUATAI SECURITES图表1：DeepSeek MoE来构求用M搬度专掌分制(Fine-grained Expert Segmentation)?以度共享◆掌(Shared E邓erts Isolation)方式Routed ExpertShared ExpertOutput HiddenOutput HiddenooOutput Hiddenoo、⑧2N-12N2N-12NRouter K=2Router K=4RouterK=3Input Hiddenloonput HiddenInput Hiddenoo(a)Conventional Top-2 Routing(b)+Fine-grained Expert Segmentation(c)+Shared Experts Isolation(DeepSeekMoE)黄未源：DeepSeek2024年1月论文《DeepSeekMoE:Towards Ultimale Expert Specialization in M对ure-of-Experts Language Models》、Medium、年泰所究图表2：MHA(Google)、GQA(Google)、MQA(Google)、MLA(DeepSeek)在KV蠖存处重中的作用示意因Cached During InferenceMulti-Head Attention (MHA)I Grouped-Query Attention(GQA)I Multi-Query Attention (MQA)I Multi-Head Latent Attention (MLA)图ValuesComoressedLatent KVQueries资未源：DeepSeek2024年5月论文《DeepSeek-V2:A Strong,Economical,and Efficient Mixture-of-Experts Language Model》Medium、年泰所究图表3：DeepSeek R1MTP(Mule-Token Prediction)示老图Head main model)Head2 (MTP Module 1)Head (MTP Module k)参数共享vocab_predict_head h x vocab_sizevocab_predict_headh x vocab_sizevocab_predict_headh x vocab_sizehxhhxhLinear Proiection2nxhLinear Proiection2h xhRMSNorm RMSNormRMSNormRMSNormast Hidden (h)Transformer Layers主干网络embeddingembeddingembedding参数共享tyTeacher foreing式l城2注：此图仅为DeepSeel欧R1模型MTP方法的示意图，尖际操作步骤可能与示意图存在差并，资科来源：DataFunTalk公众号、华泰所究：免袁声明和城露以及分斯狮声明是捉的一部分，请务必一起阅读。3