"Artificial intelligence" cannot miss the technical principle details of the mainstream large language model

outline

First, the details of the large language model

transformer and LLM

1.1 Model structure

1.2 Training objectives

1.3 tokenizer

1.4 Location Code

1.5 Layer normalization

1.6 Activation Functions

1.7 Multi-query Attention 与 Grouped-query Attention

1.8 Parallel transformer block

1.9 Summary - Training stability

Second, the distributed pre-training of LLM

Peer-to-peer communication vs. collective communication

2.1 Data parallelism

2.2 Tensor parallelism

2.3 Pipeline parallelism

2.4 3D parallelism

2.5 Mixed Precision Training

2.6 Activate Recalculation

2.7 ZeRO, zero redundancy optimizer

2.8 CPU-offload，ZeRO-offload

2.9 Flash Attention

2.10 vLLM: Paged Attention

Third, the parameters of LLM are fine-tuned efficiently

Why efficient parameter fine-tuning?

3.1 prompt tuning

3.2 prefix tuning

3.3 adapter

3.4 LLaMA adapter

3.5 LoRA

3.6 Experimental comparison