LLM 架构工作台
参数计算 + MoE 支持 + KV Cache 分析 + 推理吞吐量 + 模型对决 — 20+ 主流模型预设
2026 旗舰 MoE
2026 小 MoE / 创新架构
2024-2025 MoE
Dense (当前代)
历史馆 (过时)
总参数量
8.03B
Head Dim: 128 · GQA: 32/8
Transformer 积木图
Embedding
525.34M
6.5%
V(128.26K) × h(4096)
LM Head
525.34M
6.5%
V × h
× 32 layers
Attention
41.94M
19.2%
Q+K+V+O
MLP (SwiGLU)
176.16M
80.8%
3 × h × ffn
Norm
8.19K
0.0%
2 × h
参数分布
Embedding525.34M (6.5%)
Attention × 321.34B (16.7%)
MLP × 325.64B (70.2%)
Norm × 32262.14K (0.0%)
LM Head525.34M (6.5%)
单层: 218.11M (Attn: 41.94M + MLP: 176.16M)
推理显存
FP16/BF1616.06 GB
半精度INT88.03 GB
8位量化INT44.02 GB
4位量化GPU 适配
H100 80GBFP16
H200 141GBFP16
4090 24GBFP16
MI300X 192GBFP16
公式参考 ↓
Dense: N = Embed + L × (Attn + MLP + Norm) + LM_Head
MoE total: N = Embed + L × (Attn + Router + E×MLP + Norm) + LM_Head
MoE active: N_a = Embed + L × (Attn + Router + A×MLP + Norm) + LM_Head
Attention = Q(h×h) + K(h×kv_h×d) + V(h×kv_h×d) + O(h×h)
MLP(SwiGLU) = 3 × h × ffn
Router = h × num_experts (per layer)