Preface

I have previously written several articles on the basics of AI large language models, including introductions to local large language model UIs and API providers (see article:Starting Your AI Journey: A Detailed Introduction to Local LLM UIs and LLM API Providers), Local Large Language Model UI: Lobechat Installation and Deployment Tutorial (see article:Docker Series: Detailed Deployment Tutorial Based on the Open Source Large Language Model UI Framework: LobeChat) and Detailed Settings and Usage Tutorial (see article:Home Data Center Series: Unlocking the Full Potential of LobeChat: A Complete Guide from Setup to Practical Application). However, the API providers involved when writing these contents were basically still well-known commercial API providers, such as OpenAI (GPT series), Google (Gemini series), Anthropic (Claude series), etc.

However, in addition to commercial API providers, there are also open-source large language model frameworks that can be self-hosted locally, such as the famous Ollama. This article will use the deployment of a large language model under the Ollama framework as an example to demonstrate how to build your own open-source large language model locally.

Note: Building open-source large language models locally has high hardware requirements. Running models with larger parameter sizes requires significant memory, and if you require fast response speeds from the large language model, Nvidia graphics card support is needed, so there is a relatively high barrier to entry. Since I installed it on a mini PC with an Intel CPU and no dedicated graphics card, I can only rely on the CPU, and the response speed is hard to describe~.

Building Your Own Open-Source Large Language Model Based on Ollama

Introduction to Ollama

Ollama is a framework focused on running and managing large language models (LLMs) locally (official website address:https://ollama.com/), providing user-friendly interfaces and tools for easily loading, managing, and running various large language models. Without relying on cloud servers, Ollama allows users to run LLMs efficiently directly on their own computers, while also featuring rich extension capabilities. The core features of Ollama are as follows:

1. Local Execution: Ollama supports running large language models directly on local hardware, which is particularly attractive to data privacy-sensitive users because data is kept entirely local.
2. Model Support: Ollama is compatible with a variety of popular models, allowing users to choose and load the appropriate model and adjust it according to different task requirements. The range of models includes open-source LLMs, as well as high-quality models similar to GPT-4.
3. User-Friendly Interface: Provides an intuitive UI and CLI (Command Line Interface), allowing users to interact with models through a graphical interface or execute model commands in the terminal.
4. Easy Installation and Management: Ollama allows users to easily download, install, and update models, and manage multiple model instances, improving the usability and flexibility of local models.
5. Customization and Extension: Supports fine-tuning and parameter adjustment for various models, allowing users to optimize model performance based on specific tasks. In addition, Ollama allows local models to be used for personalized application development, enabling seamless integration into different projects.

Application scenarios for Ollama include, but are not limited to:

• Data Privacy Protection: In environments requiring high privacy and data localization, Ollama is suitable for use in enterprise, organizational, and personal development environments.

• Development and Testing: Ollama is an ideal tool for researching and testing LLMs, helping developers debug model performance locally.

• Resource Optimization: Complex models can be run without relying on cloud services, making it suitable for projects requiring local computation.

Overall, Ollama is suitable for users who want to use large language models in a local, private environment and want to flexibly manage model selection and parameters.

Local Large Language Models Supported by Ollama

Ollama supports a variety of open-source and optimized large language models, making them easy to run locally without relying on an internet connection. These models can replace commercial large language models on the market in terms of functionality and performance. The following are some common locally runnable models supported by Ollama and their applications:

LLaMA (Large Language Model Meta AI)

• Introduction: An open-source model family introduced by Meta, offering models of different scales (such as LLaMA-7B, LLaMA-13B, LLaMA-30B, etc.) with good performance and the ability to run locally. The LLaMA model architecture performs excellently under multi-task scenarios, making it suitable for common language tasks such as text generation and question answering. Among them, Llama 3 is the third-generation Llama series model open-sourced by Meta AI. Its new 8B and 70B parameter Llama 3 models achieve even greater performance improvements on top of Llama 2. Due to technical improvements in pre-training and post-training, the Llama 3 model is the best model today at the 8B and 70B parameter scales. Improvements in the Llama 3 model have significantly reduced false refusal rates, improved consistency, and increased the diversity of model responses. The Llama 3 model's capabilities in reasoning, code generation, and instruction following have also been greatly improved.

• Alternative to Commercial Models：Similar to OpenAI's GPT-3 and GPT-3.5, it is especially suitable for small projects and personal use by developers.

Mistral

• Introduction：Mistral is a performance-optimized open-source model that achieves excellent performance with less computation, enabling it to run efficiently on resource-constrained devices. Its architecture is particularly well-suited for achieving a good balance in multitasking.

• Alternative to Commercial Models：It can replace commercial models similar to the GPT-3 level, making it especially suitable for applications requiring small model deployment and low computing resources.

Falcon

• Introduction：The Falcon model is a powerful open-source model launched by TII (Technology Innovation Institute), featuring strong natural language generation and understanding capabilities. Falcon 40B performs excellently across various benchmarks, making it suitable for a variety of complex text tasks.

• Alternative to Commercial Models：It can replace commercial models at the GPT-3.5 and GPT-4 levels, with performance close to ChatGPT in some scenarios, and performs well in handling complex tasks.

Vicuna

• Introduction：This is an instruction-finetuned model based on LLaMA, specifically optimized for performance in instruction-following tasks. Vicuna's training data comes from large-scale user instruction data, making it more suitable for conversational scenarios.

• Alternative to Commercial Models：It is suitable for replacing models similar to ChatGPT or Anthropic's Claude series, performing particularly well in interactive tasks such as dialogue, Q&A, and customer support.

Code LLaMA

• Introduction：A LLaMA variant model specifically optimized by Meta for code generation tasks, supporting code completion, generation, and explanation across multiple programming languages. Designed specifically for developers, it improves the accuracy of code generation tasks.

• Alternative to Commercial Models：It is suitable for replacing GitHub Copilot or OpenAI's Codex, offering similar capabilities in code completion and programming support.

GPT-J & GPT-NeoX

• Introduction：An open-source model family developed by EleutherAI, featuring different parameter scales (such as GPT-J-6B and GPT-NeoX-20B), specifically designed to support various natural language generation tasks with excellent performance.

• Alternative to Commercial Models：GPT-J and GPT-NeoX can be used to replace commercial models at the GPT-3 level, suitable for daily conversations and generative tasks.

Dolly

• Introduction：An open-source model launched by Databricks that supports localized and customized fine-tuning, excelling at handling simple dialogue tasks and Q&A.

• Alternative to Commercial Models：It is suitable for replacing models similar to Bard or the GPT-3 level, performing exceptionally well in small projects that require personalized fine-tuning.

Bloom

• Introduction：A large-scale open-source model launched by the BigScience collaborative project, supporting multilingual processing and excelling in long-text generation tasks, with outstanding performance in multilingual applications.

• Alternative to Commercial Models：It can be used to replace commercial models such as GPT-3.5 or Claude 2, making it particularly suitable for tasks requiring multilingual support.

Summary

There are also some other models that I won't list one by one. Supported by Ollama, these models can run locally, providing alternative solutions to commercial models. They are particularly suitable for application scenarios that require data privacy, high security, customization, or are cost-sensitive. These models typically consume fewer resources than commercial models, making them suitable for users with specific tasks or customization needs.

Ollama Deployment

Deployment Methods and Differences

Ollama can be deployed either via Docker or from source. In terms of performance, there is typically no significant difference between Ollama's source deployment and Docker deployment, as the underlying model inference relies on hardware rather than the runtime environment itself. However, in specific scenarios, the following points may cause minor differences:

1. Resource Allocation：Docker containers sometimes limit direct access to hardware resources, especially under default configurations. Although Docker settings can be adjusted to provide higher resource access privileges, Docker deployment's resource management is slightly more indirect compared to running from source directly on the host.
2. System Overhead：Docker containers add a small amount of system overhead (such as storage and network layers). While these overheads are typically small, they can accumulate into minor latencies in high-performance computing scenarios.
3. Driver Compatibility：If using a GPU, source deployment can directly access local GPU drivers, whereas Docker requires dedicated containers and driver configurations with GPU support. NVIDIA provides the nvidia-docker tool to optimize Docker's GPU support, so it can typically achieve performance close to that of source deployment.

Simple summary: In environments without a GPU, there is basically no difference between source deployment and Docker deployment, and Docker deployment is more recommended in this case; if a GPU is available, source deployment is more recommended.

Docker Deployment

CPU Only

docker run --name ollama -d --restart=always --net=public-net \
-v /docker/ollama:/root/.ollama \
-p 11434:11434 \
-e OLLAMA_ORIGINS="*" \
ollama/ollama

Parameter Notes:
-v /docker/ollama:/root/.ollama: Maps ”/docker/ollama” on the host machine to ”/root/.ollama” inside the container”
-e OLLAMA_ORIGINS=”*“: Domains allowed for cross-origin access, default is ”local access”, configure as needed

Nvidia GPU

Install NVIDIA Container Toolkit (Configure on Host Machine)

Install using Apt

1. Configure Repository

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
    | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
    | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
    | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update

2. Install NVIDIA Container Toolkit

sudo apt-get install -y nvidia-container-toolkit

Install using Yum or Dnf

1. Configure Repository

curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo \
    | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

2. Install NVIDIA Container Toolkit

sudo yum install -y nvidia-container-toolkit

Configure Docker to Use Nvidia Drivers

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Create and Run Container

docker run --name ollama -d --restart=always --net=public-net \
--gpus=all \
-v ollama:/root/.ollama \
-p 11434:11434 \
-e OLLAMA_KEEP_ALIVE="10m" \
-e OLLAMA_ORIGINS="*" \
ollama/ollama

Parameter Notes:

--gpus=all: Allows the Docker container to access all Nvidia GPUs on the host machine. Enables NVIDIA GPU support, instructing Docker to allocate all available GPUs on the host machine to the container. If there are multiple GPUs on the host, this option allows the container to access all GPU devices. This option depends on the installation of ”nvidia-container-toolkit” and requires the host machine to support NVIDIA drivers. If there is no GPU or the NVIDIA support library is not installed, --gpus=all will not work, and the command will ignore GPU support.
-e OLLAMA_KEEP_ALIVE=”10m”: The duration the model remains loaded in VRAM, default is ”5m”, modify as needed

AMD GPU

To run Ollama via Docker with an AMD GPU, use the ROCm tag and the following command:

docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm

Source Deployment

Regarding the deployment operating systems, Ollama's official default source installation supports three by default: macOS, Linux, and Windows. macOS and Windows use traditional application installation methods, while Linux requires just a single installation command:

curl -fsSL https://ollama.com/install.sh | sh

The installation steps are very simple, so we won't waste space introducing them here.

---

Source installation differs from Docker installation as it requires specifying more environment variables, such as ”OLLAMA_HOST”: the bound host and port, default is ”127.0.0.1:11434", configure as needed. If the local LLM UI and Ollama are on the same device, you can use the default value; if they are not on the same device and there are remote access issues, you can try setting this parameter.

---

Install Ready-Made Model Versions

Next, run the version of the large model you need locally. For the latest model versions, parameter scales, and model sizes of the large models, you can refer to the parameters on the official GitHub page (https://github.com/ollama/ollama)：

Note: The unit ”B” here refers to the parameter scale of the model, in units of “Billion”. In other words, 1B means the model has 1 billion parameters, and 7B means it has 7 billion parameters. The number of model parameters typically reflects the complexity and capability of the model—the more parameters, the richer the linguistic features the model can capture and express, but it also requires more computing resources and memory. In large language models, parameters represent learned values such as weights and biases in the model. These parameters determine how the model processes and generates text data. Generally speaking, the larger the parameter size, the better the model's performance, but higher hardware performance support is required to run efficiently.

Different model rules have requirements for host memory:

Although the figure above only mentions 7B, 13B, and 33B, the required memory can usually be roughly estimated based on the linear proportion of the model parameter scale. For example, assuming a 7B model requires 8 GB of memory, and there is a roughly linear relationship between model parameters and memory requirements, we can infer that:
• 1B model: approx. 1.1 – 1.2 GB of memory
• 2B model: approx. 2 – 2.3 GB of memory
• 3B model: approx. 3 – 3.5 GB of memory
• 8B model: approx. 9 – 9.5 GB of memory

These are only rough estimates, and specific requirements may vary depending on the model architecture, optimization methods, or loading modes. The actual memory requirements may differ slightly based on the actual distribution and segmented loading of the model, but the overall calculation logic can serve as a reference.

Custom Models

Additional Knowledge: GGUF Model File Format

“GGUF” refers to a model file format specifically optimized for local use, particularly suitable for running LLMs (Large Language Models) on resource-constrained devices. It is a binary format that offers cross-platform and model compatibility advantages, resulting in faster loading and inference speeds. This format is becoming increasingly popular in the open-source LLM community (Reddit, Discord, and other AI community forums) because it enhances local performance without relying on cloud resources.

Some open-source model libraries also provide a large number of open-source models supporting the GGUF format. Taking ”Hugging Face” (currently one of the most popular platforms) as an example, through the Hugging Face model library, you can search for and download many pre-trained models shared by communities and organizations, which sometimes directly include the GGUF format or corresponding conversion tools.

In addition, some models supporting the GGUF format (such as open-source models like LLaMA) are directly released by development teams on platforms like GitHub. The official repositories or release pages usually list the supported formats and provide download links.

Custom Models

Ollama can create custom models by importing GGUF-formatted model files into a Modelfile. The steps are as follows.

1. First, create a file named ”Modelfile” and add the following FROM instruction to specify the local file path of the model to be imported:

 FROM ./vicuna-33b.Q4_0.gguf

2. Create a model named example in Ollama (the model name can be set arbitrarily, as you like):

ollama create example -f Modelfile

3. Run the custom model

ollama run example

Note: Modelfile is a configuration file containing the core settings and hyperparameters of the model. Common configuration options include hyperparameters such as the base source path of the model (specified via FROM), the number of layers, hidden units, and attention heads. You can also define training parameters such as the location of training data, learning rate, batch size, and optimizer. Other settings in the Modelfile include input formats, output types, device types (such as CPU or GPU), memory limits, and model metadata like author, version, and description. These parameter combinations provide flexible customization, making model loading, training, and running more efficient and adaptable to different application needs. Since there is a lot of content and I don't need to deal with it for now, I won't go into detail here. Those who need it can research it on their own.

Installing Model Llama 3.2 in Ollama and General Usage

Install Llama 3.2

Taking the ”llama3.2 3B” model as an example, since I deployed it via CPU-only Docker, the command to run this model is:

docker exec -it ollama ollama run llama3.2

Then there is a period of waiting, and the specific waiting time depends on the size of the model you choose and your network speed:

---

和command will also automatically pull the Llama 3.2 model file when Ollama finds it is not available locally, which is actually similar toThe command is similar,ollama runThe command actually includes first performing apulloperation to pull the model, and thenrun, so you can actually pull the image separately first and run it when neededollama runcommand:

docker exec -it ollama ollama pull llama3.2

---

Then you can use theollama listcommand to view the downloaded LLMs:

docker exec -it ollama ollama list

Useollama pscommand to view the running LLMs:

docker exec -it ollama ollama ps

Useollama stopcommand andlooama rmcommand combined can delete a running LLM:

docker exec -it ollama ollama stop 模型ID
docker exec -it ollama ollama rm 模型ID

You can also use the following command to seeollama -hwhat parameters the command supports:

docker exec -it ollama ollama -h

If Ollama is installed from source, you can directly run the corresponding model download and installation command in the shell:

ollama run llama3.2

Other commands are the same as under Docker, with the difference being that they can all be run directly.

You can also try other different models; the commands corresponding to specific models can be found in the image above.

---

Actually, when running the commandollama run llama3.2, the container has already entered the interactive interface with llama3.2:

However, this interactive method is not practical and is at most used for testing. To be truly practical, it still relies on a local large language model UI, such as Lobechat.

---

Adding the Llama 3.2 Model from Ollama in Lobechat

Method 1: Self-built Llama 3.2 Model (Not Recommended, Troublesome)

At the time of writing, Lobechat did not yet have built-in support for the llama 3.2 model, so it can be used by creating a custom model:

As described in my previous article (Home Data Center Series: Unlocking the Full Potential of LobeChat: A Complete Guide from Setup to Practical Application), you can create a new llama 3.2 3B model version yourself:

Then, the Llama 3.2 3B model version will appear:

Click ”Llama 3.2 3B” in the red box in the image above to enter the specific model settings page. Configure it as needed and click ”OK” to save:

Method 2: Directly Retrieve the Downloaded Model List from Ollama (Recommended)

This method is the simplest. If communication with Ollama is normal and passes the check, you can directly click the ”Get Model List” button to retrieve the models already downloaded on Ollama. For example, I downloaded two, llama3.2 and gemma2:2b, so clicking "Get Model List" directly displays them, as shown below:

Note: The only issue to pay attention to is what I annotated in the image above. If Ollama and Lobechat are on the same device, and the ”Ollama Service Address” box is left blank (indicating the use of the system defaulthttp://127.0.0.1:11434address), even though Ollama is running normally and has used theollama runcommand to enable the corresponding large model, it just won't pass the check.

If you encounter this issue, you can try changing the address from 127.0.0.1 to the actual IP address of the device's network card. It's probably a bug.

Select the Ollama 3.2 Model in the Lobechat ”Session” Interface

Directly in the chat section on the right side of Lobechat's ”Session” interface, select the corresponding model to the right of the assistant's avatar. Here I choose Llama 3.2 3B, as shown below:

Then try a simple conversation:

It immediately drove the CPU of the dedicated Docker virtual machine up to 78%:

Write a simple shell script:

The CPU of the dedicated Docker virtual machine went up to 90%~:

CPU-only deployment is still quite struggling. I tried a smaller model ”llama3.2:1b”. Compared to 3B, 1B requires fewer resources and should theoretically be faster, but well, it only feels a tiny bit faster in output speed:

The CPU utilization is slightly lower, but in reality, there isn't much difference:

Thinking about it, it's probably because the bottleneck lies in my CPU. With my crappy CPU, there's actually no difference whether it's 1B or 3B~.

Note: There are also many Chinese fine-tuned versions of llama3 on GitHub. After optimization, under the same configuration, the response speed and support for Chinese have been significantly improved. If you are interested, you can look for them by directly searching ”llama3" on GitHub. Some of the platforms and communities mentioned earlier are also search channels.

Afterword

Originally, I also wanted to study whether using Intel integrated graphics could achieve acceleration. However, after some research, it seems to make little sense because Ollama's deployment mainly supports NVIDIA GPUs due to the need for CUDA support to accelerate deep learning tasks (CUDA is NVIDIA's proprietary GPU computing platform, widely used for AI model training and inference acceleration). Intel integrated graphics lack support for CUDA. Although Intel is also developing GPU solutions suitable for AI inference (such as Intel Arc discrete graphics and its OneAPI framework), Ollama has not yet announced support for these hardwares and frameworks, let alone Intel's integrated graphics~. Even if it is worked out, it wouldn't make much sense.

However, running Ollama on Apple M-series (such as M3 Pro) can yield better performance, especially for small or medium-sized large language models (such as models with under 7B parameters): Apple's M-series chips (including M1, M2, M3 and their Pro, Max, Ultra variants) have a powerful built-in Neural Engine and GPU, specifically optimized for machine learning tasks, so their performance when running large language models locally is ”impressive”. But as for how ”impressive” it actually is, I'm not sure. My MacBook Pro is only an M1, so I really don't have much desire to test it.

It's decided, I'll buy an M4 Pro Mac mini next year to test it. I'm really going all out for research—this is dedicating myself to science, not spending money recklessly~.

Additionally, here is a simple comparison of running Ollama on Apple M-series (taking M3 Pro as an example) vs. NVIDIA GPU:

• Inference Speed

For models that are not too large (such as 7B or smaller models), the performance of the M3 Pro can approach that of entry-level NVIDIA graphics cards, and is even faster than some low-end GPUs. However, for large-scale models (such as 13B or larger), the CUDA cores and Tensor cores of professional NVIDIA GPUs (such as A100, H100, RTX 4090, etc.) can provide significant acceleration, especially excelling in video memory and parallel processing capabilities, and will therefore be faster.

• Memory Limits

The M3 Pro typically has a larger memory capacity (16GB to 36GB), which can support small and medium-sized models running in memory. However, ultra-large models (such as 33B, 65B) will be limited by memory and may not be fully loaded into memory, resulting in running failures or requiring data to be loaded in batches. In contrast, professional NVIDIA GPUs have larger video memory (such as the A100 with 40GB VRAM) and can handle larger models.

• Power Efficiency and Quietness

M-series chips like the M3 Pro excel in power efficiency and quietness, with low heat generation and low power consumption, making them suitable for long-term operation. In comparison, professional NVIDIA graphics cards have higher power consumption and heat generation, usually requiring active cooling.

Summary: Running Ollama on the M3 Pro can achieve excellent CPU and Neural Engine acceleration, making it suitable for daily use and small model inference (though the entry barrier is not low, as M3 (Pro) machines are not cheap either~; M2 (Pro) should also be able to put up a fight. In short, the better the performance, the better the experience). If you pursue higher inference speeds and smooth operation of large models, professional NVIDIA graphics cards remain the stronger choice.