前言

简单的介绍如何Local Inference Llama。

src link: https://github.com/LuYF-Lemon-love/fork-huggingface-llama-recipes

Operating System: Ubuntu 22.04.4 LTS

参考文档

  1. NLP Course - Local Inference

介绍

您想在本地运行Llama模型的推理吗？我们也是！内存需求取决于模型大小和权重的精度。下表显示了不同配置所需的大致内存：

Model Size Llama Variant BF16/FP16 FP8 INT4(AWQ/GPTQ/bnb)
1B 3.2 2.5 GB 1.25GB 0.75GB
3B 3.2 6.5 GB 3.2GB 1.75GB
8B 3.1 16 GB 8GB 4GB
70B 3.1 140 GB 70GB 35GB
405B 3.1 810 GB 405GB 204GB

这些是估计值，可能会根据特定的实现细节和优化而有所不同。

Llama-3.1-8B-Instruct in 4-bit bitsandbytes

src link: https://github.com/LuYF-Lemon-love/fork-huggingface-llama-recipes/blob/main/local_inference/4bit_bnb.ipynb

  1. 首先，我们来安装所需的库：
1
pip install transformers[torch] bitsandbytes
  1. 我们导入所需的库：
1
2
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
  1. 让我们加载模型。为了即时量化模型，我们需要传递一个 quantization_config：
1
2
3
4
5
6
7
8
9
10
11
12
from modelscope import snapshot_download

model_dir = snapshot_download('LLM-Research/Meta-Llama-3.1-8B-Instruct')

quantization_config = BitsAndBytesConfig(load_in_4bit=True,
                                         bnb_4bit_compute_dtype=torch.bfloat16,
                                         bnb_4bit_use_double_quant=True,
                                         bnb_4bit_quant_type= "nf4"
                                         )

quantized_model = AutoModelForCausalLM.from_pretrained(
	model_dir, device_map="auto", torch_dtype=torch.bfloat16, quantization_config=quantization_config)
  1. 然后，我们需要准备输入数据：
1
2
3
tokenizer = AutoTokenizer.from_pretrained(model_dir)
input_text = "What are we having for dinner?"
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
  1. 最后，我们可以生成输出！
1
2
3
output = quantized_model.generate(**input_ids, max_new_tokens=256)

print(tokenizer.decode(output[0], skip_special_tokens=True))

Llama-3.2-3B-Instruct in 8-bit bitsandbytes

src link: https://github.com/LuYF-Lemon-love/fork-huggingface-llama-recipes/blob/main/local_inference/8bit_bnb.ipynb

  1. 首先，我们来安装所需的库：
1
pip install transformers[torch] bitsandbytes
  1. 我们导入所需的库：
1
2
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
  1. 让我们加载模型。为了实时量化模型，我们传递一个量化配置：
1
2
3
4
5
6
7
from modelscope import snapshot_download

model_dir = snapshot_download('LLM-Research/Llama-3.2-3B-Instruct')
quantization_config = BitsAndBytesConfig(load_in_8bit=True)

quantized_model = AutoModelForCausalLM.from_pretrained(
	model_dir, device_map="auto", torch_dtype=torch.bfloat16, quantization_config=quantization_config)
  1. 然后，我们需要准备输入：
1
2
3
tokenizer = AutoTokenizer.from_pretrained(model_dir)
input_text = "What are we having for dinner?"
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
  1. 最后，我们可以生成输出！
1
2
3
output = quantized_model.generate(**input_ids, max_new_tokens=10)

print(tokenizer.decode(output[0], skip_special_tokens=True))

Llama-3.1-8B-Instruct with AWQ & fused ops

src link: https://github.com/LuYF-Lemon-love/fork-huggingface-llama-recipes/blob/main/local_inference/awq.ipynb

Model Checkpoint: https://huggingface.co/hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4

注意：虽然在这个例子中我们只使用了8B的Instruct检查点，但您可以使用相同的代码库来处理任何Llama 3.1模型的检查点，例如70B、405B（以及微调）！

  1. 由于Llama 3.1带来了一些小的模型变化（主要是RoPE缩放），我们需要确保我们使用的是transformers的最新版本。
1
pip install -q --upgrade transformers autoawq accelerate
  1. 加载分词器和模型检查点。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, AwqConfig

model_id = "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4"

quantization_config = AwqConfig(
    bits=4,
    fuse_max_seq_len=512, # Note: Update this as per your use-case
    do_fuse=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
  model_id,
  torch_dtype=torch.float16,
  low_cpu_mem_usage=True,
  quantization_config=quantization_config
).to("cuda")
  1. 定义提示并进行分词处理。
1
2
3
4
5
6
7
8
9
10
11
12
prompt = [
  {"role": "system", "content": "You are a helpful assistant, that responds as a pirate."},
  {"role": "user", "content": "What's Deep Learning?"},
]

inputs = tokenizer.apply_chat_template(
  prompt,
  tokenize=True,
  add_generation_prompt=True,
  return_tensors="pt",
  return_dict=True,
).to("cuda")
  1. 生成。
1
2
outputs = model.generate(**inputs, do_sample=True, max_new_tokens=25)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

瞧！现在你拥有了一个智能且能干的助手！🦙

Llama-3.1-8B-Instruct in INT4 with GPTQ

src link: https://github.com/LuYF-Lemon-love/fork-huggingface-llama-recipes/blob/main/local_inference/gptq_generation.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# INSTALLATION
# pip install -q --upgrade transformers accelerate optimum
# pip install -q --no-build-isolation auto-gptq
# REQUIREMENTS
# An instance with at least ~210 GiB of total GPU memory when using the 405B model.
# The INT4 versions of the 70B and 8B models require ~35 GiB and ~4 GiB, respectively.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "hugging-quants/Meta-Llama-3.1-8B-Instruct-GPTQ-INT4"
messages = [
    {"role": "system", "content": "You are a pirate"},
    {"role": "user", "content": "What's Deep Leaning?"},
]

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
    device_map="auto",
)

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to("cuda")

outputs = model.generate(**inputs, do_sample=True, max_new_tokens=256)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

结语

第一百九十九篇博文写完，开心！！！！

今天，也是充满希望的一天。