Chat-GPT原理

GPT原理

核心是基于Transformer 架构

英文原文：

​ Transformers are based on the “attention mechanism,” which allows the model to pay more attention to some inputs than others, regardless of where they show up in the input sequence. For example, let’s consider the following sentence:

​ In this scenario, when the model is predicting the verb “bought,” it needs to match the past tense of the verb “went.” In order to do that, it has to pay a lot of attention to the token “went.” In fact, it may pay more attention to the token “went” than to the token “and,” despite the fact that “went” appears much earlier in the input sequence.

​ 它允许模型在处理输入序列时能够同时关注输入序列中各个位置的信息，从而更好地捕捉长距离依赖关系。

​ Transformer架构通常由编码器（encoder）和解码器（decoder）组成，其中编码器用于将输入序列映射为一系列隐藏表示，解码器则利用这些隐藏表示生成输出序列。每个编码器和解码器由多个相同的层（layer）堆叠而成，每个层都包含自注意力子层和前馈神经网络子层。

​ 在自注意力子层中，输入序列中的每个元素都可以与其他元素进行交互，通过学习注意力权重来确定不同位置之间的关联程度。这种机制使得模型能够更好地处理长距离依赖关系，从而在翻译、摘要生成等任务中取得了很好的效果。

​ 除了自注意力机制外，Transformer架构还使用了残差连接（residual connections）和层归一化（layer normalization）等技术来加速训练过程和提高模型性能。此外，Transformer架构还支持并行计算，使得模型能够更高效地处理大规模数据。

原文链接建议多读读：How GPT Models Work. Learn the core concepts behind OpenAI’s… | by Beatriz Stollnitz | Towards Data Science