Chat with DeepSeek Using OpenVINO

Recently, DeepSeek has been very popular. I just bought a new laptop with a Lunarlake CPU: the Intel® Core™ Ultra 7 Processor 258V. This CPU comes with an integrated NPU. If I were to naively use only the CPU to run DeepSeek, the performance would be quite poor.

Initial Attempt - Running DeepSeek on CPU

Running DeepSeek on a CPU is not difficult. First, install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Then pull down the DeepSeek model:

ollama pull deepseek-r1:32b

The 32B model consumes approximately 22GB of memory. My 32GB of RAM is just enough to handle it.

With that set up, you can proceed with normal conversations. You can use Open WebUI to set up a simple Web service:

pipx install open-webui
open-webui serve

Simply open your browser and navigate to http://localhost:8080 to see a simple web chat interface.

Obviously, the Lunarlake CPU is least suited for multi-core computing. DeepSeek's model is a very large model; running it on my laptop is more of a novelty than anything practical. It has no real-world value.

OpenVINO

OpenVINO (Open Visual Inference and Neural Network Optimization) is an open-source toolkit developed by Intel, designed to accelerate the training and inference of artificial intelligence (AI) and machine learning models. Its goal is to help developers efficiently deploy AI applications on Intel hardware, such as CPUs, GPUs, and VPUs.

  1. OpenVINO supports converting models trained with various deep learning frameworks (such as TensorFlow, PyTorch, ONNX, etc.) into an Intermediate Representation (IR) and optimizes them to improve performance.
  2. OpenVINO can run on both cloud and edge devices, making it suitable for a wide range of scenarios, including computer vision and natural language processing.
  3. OpenVINO significantly improves model inference speed through hardware acceleration, such as Intel CPUs, NPUs, and graphics cards.

Installing NPU Drivers

Initially, I installed OpenVINO directly because I was unaware that the NPU driver needed to be installed first. However, I found that OpenVINO could not detect the NPU. It was only later that I learned the NPU driver must be installed.

I am using AnduinOS 1.2.0, which is equivalent to Ubuntu 24.10. First, go to https://github.com/intel/linux-npu-driver/releases to download the latest driver.

wget https://github.com/intel/linux-npu-driver/releases/download/v1.13.0/intel-driver-compiler-npu_1.13.0.20250131-13074932693_ubuntu24.04_amd64.deb
wget https://github.com/intel/linux-npu-driver/releases/download/v1.13.0/intel-fw-npu_1.13.0.20250131-13074932693_ubuntu24.04_amd64.deb
wget https://github.com/intel/linux-npu-driver/releases/download/v1.13.0/intel-level-zero-npu_1.13.0.20250131-13074932693_ubuntu24.04_amd64.deb

The three packages above are the driver, firmware, and Level Zero. The links above may no longer be the latest; do not copy and paste them. Instead, visit the official website to find the most up-to-date links.

In addition, we also need to install TBB:

sudo apt update
sudo apt install libtbb12

Don't forget to install these three packages:

sudo dpkg -i *.deb

At this point, I found that level-zero was not installed. I need to install it manually:

wget https://github.com/oneapi-src/level-zero/releases/download/v1.18.5/level-zero_1.18.5+u24.04_amd64.deb
sudo dpkg -i level-zero*.deb

Finally, we need to restart the computer:

sudo reboot

After the reboot is complete, we will see that the NPU driver has been installed. It provides a file: /dev/accel/accel0.

However, by default, only the root user can access this file. We need to add the current user to the render group:

# set the render group for accel device
sudo chown root:render /dev/accel/accel0
sudo chmod g+rw /dev/accel/accel0
# add user to the render group
sudo usermod -a -G render $USER
# user needs to restart the session to use the new group (log out and log in)

Don't forget to log out and log back in.

You can run cat on /dev/accel/accel0 to check for any output. If there is output, it indicates that the NPU driver was installed successfully.

cat /dev/accel/accel0

file

Install OpenVINO

The OpenVINO frontend is written in Python, and the backend is written in C++. We can use Python to install and use OpenVINO.

Obviously, before getting started, you need Python3 and pip:

sudo apt update
sudo apt install python3 python3-pip

Next, I use python virtualenv to create a virtual environment. Considering that my device is too new, I directly use the Nightly version:

python3 -m venv openvino_nightly_env
source openvino_nightly_env/bin/activate

Next, we can install OpenVINO:

export CXXFLAGS="-include cstdint"
pip install --pre -U openvino-genai --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly

Next, install the dependencies for OpenVINO, including ONNX and Optimum.

pip install -U onnx
pip install -U openvino
pip install -U "optimum[openvino]"

file

The square brackets in the syntax "optimumopenvino" represent the extras syntax for Python packages, used to specify optional dependencies or additional features. In other words, this installs the optimum package itself while also installing the optional dependencies related to openvino.

Using OpenVINO

Similar to CUDA, after installing OpenVINO, we also need to first verify whether OpenVINO can find the NPU.

In the virtual environment we just created, we can use Python to verify this. Run python, then enter the following code:

# 检查可用设备
from openvino import Core
core = Core()
print("Available devices:", core.available_devices)
# 输出中应包含你期望使用的设备（例如 "NPU"）

On my end, the output is:

['CPU', 'GPU', 'NPU']

file

This indicates that OpenVINO has detected the NPU.

Next, we need to find a model suitable for my NPU. My NPU only supports 4-bit models. I found a suitable model: AIFunOver/DeepSeek-R1-Distill-Qwen-32B-openvino-4bit.

Its model repository is located at https://huggingface.co/AIFunOver/DeepSeek-R1-Distill-Qwen-32B-openvino-4bit.

According to its description:

This model is a quantized version of deepseek-ai/DeepSeek-R1-Distill-Qwen-32B and is converted to the OpenVINO format. This model was obtained via the nncf-quantization space with optimum-intel. First make sure you have optimum-intel installed:

It has already been quantized for us and is compatible with OpenVINO. We can use this model directly.

Of course, if you prefer not to use a quantized model from Hugging Face, you can also convert it to OpenVINO yourself: https://docs.openvino.ai/2024/openvino-workflow/model-preparation/convert-model-to-ir.html

from transformers import AutoTokenizer
from optimum.intel import OVModelForCausalLM

import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning)

# 模型和 tokenizer 的名称
model_id = "AIFunOver/DeepSeek-R1-Distill-Qwen-32B-openvino-4bit"

model = OVModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
# 设置 pad_token 为 eos_token 以消除警告
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# 配置模型使用 NPU，并编译模型
model.ov_config["device"] = "NPU"
compiled_model = model.compile()

print("对话开始，输入 'exit' 或 'quit' 退出。")

while True:
    prompt = input("用户: ").strip()
    if prompt.lower() in ("exit", "quit"):
        break

    inputs = tokenizer(prompt, return_tensors="pt")
    generated_ids = model.generate(
        inputs["input_ids"],
        attention_mask=inputs["attention_mask"],
        max_new_tokens=5000
    )
    response = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
    print("模型:", response)

This code starts a conversation, and you can now chat with DeepSeek.

After testing, it is much faster than CPU. (Though honestly, it's not that much faster.)

file