Multilingual LLAMA Bot General (m-llama-bot-general)
Method
- Promt-Tuning/Adapter-tuning
- Parameters
| Parameters | No. |
|---|---|
| All | 6739645440 (100.00%) |
| Trainable | 6537216 (0.02%) |
| Freezed | 1229824 (99.98%) |
<img src="https://huggingface.co/anhdungitvn/m-llama-bot-general-7b/resolve/main/adapters.png" width="800">
Model
Adapter
LAYER NAME #PARAMS RATIO MEM(MB)
--model: 1,229,824 100.00% 4.69
--layers: 1,024 0.08% 0.00
--0
--attention
--gate: 32 0.00% 0.00
--1
--attention
--gate: 32 0.00% 0.00
--2
--attention
--gate: 32 0.00% 0.00
--3
--attention
--gate: 32 0.00% 0.00
--4
--attention
--gate: 32 0.00% 0.00
--5
--attention
--gate: 32 0.00% 0.00
--6
--attention
--gate: 32 0.00% 0.00
--7
--attention
--gate: 32 0.00% 0.00
--8
--attention
--gate: 32 0.00% 0.00
--9
--attention
--gate: 32 0.00% 0.00
--10
--attention
--gate: 32 0.00% 0.00
--11
--attention
--gate: 32 0.00% 0.00
--12
--attention
--gate: 32 0.00% 0.00
--13
--attention
--gate: 32 0.00% 0.00
--14
--attention
--gate: 32 0.00% 0.00
--15
--attention
--gate: 32 0.00% 0.00
--16
--attention
--gate: 32 0.00% 0.00
--17
--attention
--gate: 32 0.00% 0.00
--18
--attention
--gate: 32 0.00% 0.00
--19
--attention
--gate: 32 0.00% 0.00
--20
--attention
--gate: 32 0.00% 0.00
--21
--attention
--gate: 32 0.00% 0.00
--22
--attention
--gate: 32 0.00% 0.00
--23
--attention
--gate: 32 0.00% 0.00
--24
--attention
--gate: 32 0.00% 0.00
--25
--attention
--gate: 32 0.00% 0.00
--26
--attention
--gate: 32 0.00% 0.00
--27
--attention
--gate: 32 0.00% 0.00
--28
--attention
--gate: 32 0.00% 0.00
--29
--attention
--gate: 32 0.00% 0.00
--30
--attention
--gate: 32 0.00% 0.00
--31
--attention
--gate: 32 0.00% 0.00
--adapter_query
--weight: 1,228,800 99.92% 4.69
Model
LAYER NAME #PARAMS RATIO MEM(MB)
--model: 6,739,645,440 100.00% 12854.85
--tok_embeddings
--weight: 131,072,000 1.94% 250.00
--layers: 6,476,268,544 96.09% 12352.50
--0: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--1: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--2: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--3: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--4: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--5: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--6: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--7: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--8: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--9: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--10: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--11: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--12: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--13: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--14: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--15: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--16: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--17: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--18: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--19: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--20: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--21: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--22: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--23: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--24: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--25: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--26: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--27: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--28: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--29: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--30: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--31: 202,383,392 3.00% 386.02
--attention: 67,108,896 1.00% 128.00
--gate: 32 0.00% 0.00
--wq: 16,777,216 0.25% 32.00
--wk: 16,777,216 0.25% 32.00
--wv: 16,777,216 0.25% 32.00
--wo: 16,777,216 0.25% 32.00
--feed_forward: 135,266,304 2.01% 258.00
--w1: 45,088,768 0.67% 86.00
--w2: 45,088,768 0.67% 86.00
--w3: 45,088,768 0.67% 86.00
--attention_norm
--weight: 4,096 0.00% 0.01
--ffn_norm
--weight: 4,096 0.00% 0.01
--norm
--weight: 4,096 0.00% 0.01
--output
--weight: 131,072,000 1.94% 250.00
--adapter_query
--weight: 1,228,800 0.02% 2.34
References
- LLAM: <a href="https://ai.facebook.com/blog/large-language-model-llama-meta-ai/" download>LLaMA: A foundational, 65-billion-parameter large language model</a>
- Prompt Tuning: <a href="https://arxiv.org/abs/2104.08691" download>The Power of Scale for Parameter-Efficient Prompt Tuning</a>
- Prompt Tuning v2: <a href="https://arxiv.org/abs/2110.07602" download>P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks</a>